Oct 30 05:52:23.424320 ip-10-0-128-31 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:52:23.798674 ip-10-0-128-31 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:52:23.798674 ip-10-0-128-31 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:52:23.798674 ip-10-0-128-31 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:52:23.798674 ip-10-0-128-31 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:52:23.803243 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803165 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:23.803243 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803242 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803247 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803250 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803252 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803255 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803258 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803261 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803263 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803266 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803268 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803271 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803273 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803276 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803278 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803281 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803283 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803286 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803288 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803291 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:23.803308 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803293 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803296 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803304 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803307 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803310 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803312 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803315 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803318 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803321 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803324 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803326 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803329 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803331 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803334 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803336 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803339 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803341 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803344 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803346 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:23.803742 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803348 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803351 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803353 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803356 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803358 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803360 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803363 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803366 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803368 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803371 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803373 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803375 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803380 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803383 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803387 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803390 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803393 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803395 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803398 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803400 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:23.804199 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803402 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803405 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803408 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803410 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803413 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803415 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803417 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803420 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803422 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803424 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803427 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803429 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803432 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803434 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803437 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803439 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803443 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803446 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803449 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:23.804672 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803452 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803455 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803457 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803460 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803462 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803465 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803467 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803471 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803474 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803476 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803479 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803482 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803484 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803487 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803489 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803492 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803494 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803497 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803499 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:23.805124 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803502 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803504 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803507 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803510 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803512 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803515 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803518 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803520 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.803523 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804273 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804280 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804282 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804286 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804289 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804291 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804294 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804296 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804299 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804302 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804305 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:23.805579 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804308 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804311 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804314 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804316 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804319 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804322 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804324 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804327 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804329 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804332 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804335 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804339 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804342 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804344 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804347 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804349 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804352 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804354 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804357 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:23.806182 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804359 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804362 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804364 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804366 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804369 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804371 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804381 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804384 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804386 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804388 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804391 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804394 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804396 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804399 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804402 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804404 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804407 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804409 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804412 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804415 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:23.806640 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804417 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804420 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804422 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804425 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804427 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804430 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804432 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804435 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804437 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804440 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804442 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804445 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804447 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804449 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804452 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804454 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804457 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804460 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804462 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:23.807132 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804466 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804471 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804474 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804476 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804479 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804481 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804484 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804486 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804489 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804491 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804494 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804496 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804499 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804502 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804505 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804507 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804509 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804512 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804515 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804517 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:23.807589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804520 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804522 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804525 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804527 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804530 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804532 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804535 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804537 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804540 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804543 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804546 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804548 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804551 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804555 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804558 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804560 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.804563 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805031 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805036 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:23.808081 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805039 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805041 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805044 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805062 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805064 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805067 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805069 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805072 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805075 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805079 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805083 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805086 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805089 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805091 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805094 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805096 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805099 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805101 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805104 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:23.808539 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805106 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805109 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805111 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805114 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805116 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805119 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805121 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805125 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805128 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805131 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805133 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805136 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805139 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805141 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805144 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805146 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805149 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805151 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805154 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805156 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:23.808971 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805158 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805161 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805163 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805166 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805168 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805171 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805173 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805175 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805178 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805180 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805183 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805186 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805188 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805191 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805193 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805196 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805198 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805201 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805204 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805207 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:23.809497 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805209 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805212 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805214 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805217 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805219 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805222 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805224 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805226 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805229 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805231 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805233 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805236 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805239 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805242 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805245 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805247 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805250 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805252 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805254 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805257 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:23.809969 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805259 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805262 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805264 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805266 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805269 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805271 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805274 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805276 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805279 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805281 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805284 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805287 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805289 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805292 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805294 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805297 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805300 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805302 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805304 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:23.810457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805307 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805310 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805313 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805315 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805318 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.805320 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807008 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807023 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807029 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807034 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807038 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807041 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807058 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807063 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807066 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807069 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807072 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807076 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807079 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807082 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807087 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807090 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807093 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:52:23.810894 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807096 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807100 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807103 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807106 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807110 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807114 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807117 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807120 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807123 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807126 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807129 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807132 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807135 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807139 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807141 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807146 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807149 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807152 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807154 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807157 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807160 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807164 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807167 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807170 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:52:23.811467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807173 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807176 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807180 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807183 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807186 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807190 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807193 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807196 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807199 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807202 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807204 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807207 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807210 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807214 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807217 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807220 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807223 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807226 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807229 2578 flags.go:64] FLAG: --help="false" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807232 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-128-31.ec2.internal" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807235 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807238 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807240 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807244 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:52:23.812100 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807247 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807249 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807252 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807255 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807258 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807260 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807264 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807266 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807269 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807272 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807275 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807277 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807280 2578 flags.go:64] FLAG: --lock-file="" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807283 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807286 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807288 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807294 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807296 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807299 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807302 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807305 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807308 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807311 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807314 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807318 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:52:23.812664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807321 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807325 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807328 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807330 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807333 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807336 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807339 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807342 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807344 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807351 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807354 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807357 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807360 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807363 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807366 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807369 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807372 2578 flags.go:64] FLAG: --port="10250" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807375 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807378 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01051a8400d4b8709" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807381 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807384 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807386 2578 flags.go:64] FLAG: --register-node="true" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807389 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807393 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807395 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807398 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:52:23.813333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807401 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807405 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807408 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807411 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807414 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807416 2578 flags.go:64] FLAG: --runonce="false" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807419 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807422 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807425 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807427 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807430 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807433 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807436 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807439 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807442 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807444 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807447 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807450 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807453 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807456 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807458 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807464 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807466 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807469 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807473 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:52:23.813951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807475 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807478 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807481 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807483 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807486 2578 flags.go:64] FLAG: --v="2" Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807490 2578 flags.go:64] FLAG: --version="false" Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807494 2578 flags.go:64] FLAG: --vmodule="" Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807498 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807501 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807633 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807636 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807639 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807643 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807645 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807651 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807654 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807657 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807659 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807661 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807664 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807666 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807669 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:23.814540 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807672 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807674 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807678 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807682 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807686 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807690 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807693 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807695 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807698 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807701 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807704 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807706 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807709 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807711 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807713 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807716 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807718 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807721 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807723 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807726 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:23.815061 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807728 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807731 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807733 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807736 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807739 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807741 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807744 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807746 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807749 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807751 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807753 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807756 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807758 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807761 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807764 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807768 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807771 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807773 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807776 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807778 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:23.815533 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807780 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807783 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807785 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807788 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807790 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807792 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807795 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807797 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807800 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807802 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807804 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807807 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807809 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807812 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807814 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807816 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807819 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807822 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807824 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:23.816004 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807827 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807831 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807834 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807837 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807841 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807844 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807847 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807850 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807854 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807857 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807860 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807862 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807864 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807867 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807869 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807872 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807875 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807877 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807880 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:23.816469 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807883 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807886 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807888 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807891 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807893 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807896 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807898 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807901 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807903 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807906 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807908 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807911 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807913 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807916 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807918 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.807943 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:52:23.816959 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807977 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807981 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807985 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807988 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807993 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.807996 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808000 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808003 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808006 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808009 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808012 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808015 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808018 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808021 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808024 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808027 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808030 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808032 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808035 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:23.817647 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808037 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808040 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808043 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808057 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808062 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808064 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808067 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808070 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808072 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808075 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808078 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808080 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808083 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808086 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808089 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808091 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808094 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808098 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808100 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:23.818142 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808104 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808106 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808109 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808111 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808113 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808116 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808118 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808121 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808123 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808126 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808128 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808131 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808133 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808136 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808138 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808141 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808144 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808148 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808151 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808154 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:23.818590 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808156 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808158 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808161 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808164 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808166 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808169 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808171 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808174 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808176 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808178 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808182 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808184 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808188 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808190 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808193 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808195 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808198 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808200 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808202 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808205 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:23.819073 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808207 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808210 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808212 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808215 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808217 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808219 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808222 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808225 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808227 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808230 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808232 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808235 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808237 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808239 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808242 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808244 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808247 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808250 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808252 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:23.819591 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808254 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808257 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808260 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808263 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808266 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808269 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808272 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808274 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.808277 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.814727 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.814739 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.814756 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.814760 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814816 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814820 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814823 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814826 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814829 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814831 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:23.820062 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814834 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814836 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814839 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814842 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814844 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814847 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814849 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814851 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814854 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814856 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814860 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814862 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814865 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814867 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814870 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814872 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814875 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814878 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814880 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:23.820549 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814883 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814885 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814888 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814890 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814892 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814895 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814897 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814900 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814902 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814905 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814907 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814910 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814912 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814914 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814916 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814919 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814921 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814924 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814927 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:23.821000 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814931 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814934 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814937 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814940 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814943 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814946 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814949 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814951 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814954 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814956 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814959 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814961 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814964 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814966 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814969 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814971 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814974 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814976 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814979 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814981 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:23.821457 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814983 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814986 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814988 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814991 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814995 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.814998 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815001 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815004 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815006 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815009 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815012 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815014 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815017 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815019 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815022 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815024 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815027 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815029 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815032 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:23.821954 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815034 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815037 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815039 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815042 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815057 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815060 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815063 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815065 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815068 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815071 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815073 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815076 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815078 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815081 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815084 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815086 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815088 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815091 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815093 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815095 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:23.822567 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815098 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815100 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815103 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.815128 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815263 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815267 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815270 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815273 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815275 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815277 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815280 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815282 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815285 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815288 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815290 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815293 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:23.823029 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815295 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815298 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815300 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815303 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815305 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815307 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815310 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815312 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815314 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815318 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815321 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815324 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815327 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815329 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815331 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815334 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815336 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815339 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815341 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:23.823466 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815343 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815346 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815348 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815350 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815353 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815355 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815358 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815360 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815362 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815365 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815367 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815371 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815373 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815376 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815378 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815381 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815383 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815386 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815388 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815390 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:23.823901 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815393 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815395 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815398 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815400 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815402 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815405 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815407 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815409 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815412 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815414 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815416 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815419 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815422 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815424 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815426 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815429 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815431 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815433 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815436 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:23.824399 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815438 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815440 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815443 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815445 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815449 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815451 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815454 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815457 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815459 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815461 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815464 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815466 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815468 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815471 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815473 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815476 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815478 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815480 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815483 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:23.824906 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815485 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815487 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815489 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815492 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815494 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815497 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815499 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815502 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815506 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815508 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815512 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815516 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815519 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815522 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815525 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815527 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:23.825481 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815530 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.815552 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815585 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815589 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815592 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815595 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815597 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815600 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815602 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815605 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815607 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815610 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815612 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815614 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815617 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815620 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:23.825868 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815622 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815625 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815627 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815630 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815632 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815634 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815637 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815640 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815643 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815646 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815650 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815653 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815655 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815658 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815660 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815663 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815665 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815668 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815670 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:23.826329 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815673 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815675 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815678 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815680 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815683 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815685 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815688 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815690 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815693 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815695 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815698 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815700 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815702 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815705 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815707 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815710 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815712 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815716 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815719 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:23.826815 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815723 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815725 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815730 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815732 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815736 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815739 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815741 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815744 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815746 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815749 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815751 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815753 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815756 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815758 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815761 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815764 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815766 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815769 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815772 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815774 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:23.827485 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815776 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815779 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815781 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815784 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815786 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815789 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815793 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815796 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815798 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815801 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815804 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815806 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815808 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815811 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815814 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815817 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815821 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815823 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815826 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:23.827955 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815828 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815831 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815834 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815836 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815839 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815841 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815843 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815846 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815849 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815852 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815854 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815857 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815859 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815861 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:23.815864 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.815911 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.819024 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.819926 2578 server.go:1014] "Starting client certificate rotation" Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.820014 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:52:23.828420 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.820772 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:52:23.828856 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.821918 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:52:23.828856 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.823008 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:52:23.832624 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.832608 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:52:23.835740 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.835726 2578 log.go:25] "Validated CRI v1 image API" Oct 30 05:52:23.836570 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.836545 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:52:23.838815 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.838799 2578 fs.go:135] Filesystem UUIDs: map[4bfed4fc-9bbf-4f82-a1a5-899e02372e24:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d525d619-e4c4-4133-a33e-9afb59a77725:/dev/nvme0n1p4] Oct 30 05:52:23.838855 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.838816 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 05:52:23.843804 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.843692 2578 manager.go:217] Machine: {Timestamp:2026-10-30 05:52:23.842473981 +0000 UTC m=+0.306463501 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101902 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26e662cb2e5378ea6af81a4ea81ff5 SystemUUID:ec26e662-cb2e-5378-ea6a-f81a4ea81ff5 BootID:879fd1ee-2deb-41d8-a7b0-a29c1bcc8fff Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:8e:27:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:8e:27:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:5a:bd:a8:25:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:52:23.843804 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.843803 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:52:23.843902 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.843875 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:52:23.846445 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.846423 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:52:23.846577 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.846446 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:52:23.846623 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.846579 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:52:23.846623 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.846587 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:52:23.846673 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.846623 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:52:23.846673 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.846642 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:52:23.847350 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.847340 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:52:23.849817 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.849806 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:52:23.849940 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.849930 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:52:23.850650 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.850641 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:52:23.850685 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.850658 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:52:23.850685 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.850668 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:52:23.850685 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.850682 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:52:23.850770 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.850690 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:52:23.851104 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.851083 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:52:23.855708 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.855672 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:52:23.862931 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.862915 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:52:23.863725 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.863712 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:52:23.863767 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.863746 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:52:23.863899 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.863890 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:52:23.863935 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.863907 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:52:23.863935 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.863913 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:52:23.863935 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.863918 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:52:23.863935 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.863923 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:52:23.863935 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.863928 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:52:23.864401 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.864393 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:52:23.864430 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.864403 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:52:23.864430 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.864410 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:52:23.864430 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.864416 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:52:23.864430 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.864424 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:52:23.864581 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.864433 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:52:23.864921 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.864913 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:52:23.864952 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.864922 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:52:23.868701 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.868688 2578 server.go:1263] "Started kubelet" Oct 30 05:52:23.868789 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.868747 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:52:23.868855 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.868811 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:52:23.868909 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.868883 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:52:23.869402 ip-10-0-128-31 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:52:23.870124 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.870016 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:52:23.871171 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.871155 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:52:23.875342 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.875321 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:52:23.875420 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.875382 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 05:52:23.876467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.876452 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:52:23.876581 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.876560 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:23.877032 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877013 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:52:23.877131 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877070 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:52:23.877131 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877016 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:52:23.877273 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877158 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:52:23.877273 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877168 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:52:23.877331 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877320 2578 factory.go:55] Registering systemd factory Oct 30 05:52:23.877367 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877333 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:52:23.877556 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877543 2578 factory.go:153] Registering CRI-O factory Oct 30 05:52:23.877601 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877559 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 05:52:23.877638 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877605 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:52:23.877638 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877620 2578 factory.go:103] Registering Raw factory Oct 30 05:52:23.877713 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.877641 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 05:52:23.878679 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.878665 2578 manager.go:319] Starting recovery of all containers Oct 30 05:52:23.891836 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.891818 2578 manager.go:324] Recovery completed Oct 30 05:52:23.896458 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.896445 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:23.898069 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.898036 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:23.898135 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.898085 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:23.898135 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.898098 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:23.898595 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.898583 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:52:23.898595 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.898593 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:52:23.898687 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.898609 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:52:23.898687 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.898632 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:23.899362 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.899346 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:23.899425 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.899375 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:23.899425 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.899384 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:23.899690 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.899680 2578 policy_none.go:50] "Start" Oct 30 05:52:23.899723 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.899695 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:52:23.899723 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.899705 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:52:23.900625 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.900615 2578 policy_none.go:44] "Start" Oct 30 05:52:23.903773 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.903745 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 05:52:23.903773 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.903746 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 05:52:23.904710 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.903737 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18e338d049f16df2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-10-30 05:52:23.86850149 +0000 UTC m=+0.332491007,LastTimestamp:2026-10-30 05:52:23.86850149 +0000 UTC m=+0.332491007,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}" Oct 30 05:52:23.924386 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.924367 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 05:52:23.924455 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.924377 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18e338d04bb49a69 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-31.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-10-30 05:52:23.898069609 +0000 UTC m=+0.362059127,LastTimestamp:2026-10-30 05:52:23.898069609 +0000 UTC m=+0.362059127,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}" Oct 30 05:52:23.924645 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.924623 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 05:52:23.924645 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.924636 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 05:52:23.927617 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.927589 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:52:23.931369 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.931357 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 05:52:23.931435 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.931409 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:52:23.931435 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.931418 2578 server.go:87] "Starting device plugin registration server" Oct 30 05:52:23.931615 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.931605 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:52:23.931648 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.931615 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:52:23.931742 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.931723 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:52:23.931824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.931807 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:52:23.931824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.931823 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:52:23.932292 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.932277 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:52:23.932361 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.932315 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:23.946276 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.946258 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:52:23.953527 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.953463 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18e338d04bb4f170 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-31.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-10-30 05:52:23.898091888 +0000 UTC m=+0.362081406,LastTimestamp:2026-10-30 05:52:23.898091888 +0000 UTC m=+0.362081406,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}" Oct 30 05:52:23.971854 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.971832 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n9ldl" Oct 30 05:52:23.979654 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:23.979575 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18e338d04bb51a47 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-31.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-10-30 05:52:23.898102343 +0000 UTC m=+0.362091861,LastTimestamp:2026-10-30 05:52:23.898102343 +0000 UTC m=+0.362091861,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}" Oct 30 05:52:23.988630 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:23.986080 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n9ldl" Oct 30 05:52:24.031695 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.031676 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:24.032740 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.032604 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:24.032846 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.032758 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:24.032846 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.032772 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:24.032846 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.032814 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.033440 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.033416 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:52:24.034775 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.034761 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:52:24.034842 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.034780 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:52:24.034842 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.034796 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:52:24.034842 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.034829 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 05:52:24.049204 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.049152 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:24.053075 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.053042 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.053167 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.053084 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-31.ec2.internal\": node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:24.091946 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.091922 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:24.135910 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.135889 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"] Oct 30 05:52:24.135972 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.135950 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:24.136837 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.136819 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:24.136929 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.136846 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:24.136929 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.136858 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:24.138298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.138261 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:24.138423 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.138408 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.138472 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.138439 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:24.138982 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.138969 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:24.138982 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.138978 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:24.139117 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.138997 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:24.139117 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.139001 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:24.139117 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.139009 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:24.139117 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.139012 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:24.140171 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.140158 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.140225 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.140182 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:24.140834 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.140819 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:24.140903 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.140851 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:24.140903 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.140863 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:24.166935 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.166918 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-31.ec2.internal\" not found" node="ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.170481 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.170463 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-31.ec2.internal\" not found" node="ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.192860 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.192840 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:24.278111 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.278083 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.278111 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.278112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.278291 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.278127 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.293179 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.293144 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:24.379095 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.379031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.379095 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.379071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.379095 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.379087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.379280 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.379114 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.379280 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.379125 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.379280 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.379139 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.393604 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.393577 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:24.469911 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.469879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.473659 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.473639 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:52:24.494483 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.494457 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:24.595421 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.595370 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:24.695919 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.695842 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:24.796404 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.796373 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:24.819549 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.819521 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 05:52:24.819969 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.819681 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 05:52:24.897456 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.897426 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:24.990508 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.990431 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:47:25 +0000 UTC" deadline="2028-06-13 20:04:47.531926775 +0000 UTC" Oct 30 05:52:24.990508 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:24.990467 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14222h12m22.54146254s" Oct 30 05:52:24.997566 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:24.997531 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:52:25.021886 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:25.021853 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded322e80a89b2bbeafbf0f88fe9597d6.slice/crio-5264bed43ef0b66623dec4796da7b3bb76bfbb97bffd3026398a3dccb6a16e40 WatchSource:0}: Error finding container 5264bed43ef0b66623dec4796da7b3bb76bfbb97bffd3026398a3dccb6a16e40: Status 404 returned error can't find the container with id 5264bed43ef0b66623dec4796da7b3bb76bfbb97bffd3026398a3dccb6a16e40 Oct 30 05:52:25.022028 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:25.021889 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:25.022110 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:25.022089 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb6269644393a710c6775a446a340ab.slice/crio-f4dbbaf4e2c1247524a00d3c23a0803587d49d72584cd287c53b4e9200982013 WatchSource:0}: Error finding container f4dbbaf4e2c1247524a00d3c23a0803587d49d72584cd287c53b4e9200982013: Status 404 returned error can't find the container with id f4dbbaf4e2c1247524a00d3c23a0803587d49d72584cd287c53b4e9200982013 Oct 30 05:52:25.027906 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:25.027889 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:52:25.037948 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:25.037912 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerStarted","Data":"f4dbbaf4e2c1247524a00d3c23a0803587d49d72584cd287c53b4e9200982013"} Oct 30 05:52:25.038877 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:25.038859 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" event={"ID":"ed322e80a89b2bbeafbf0f88fe9597d6","Type":"ContainerStarted","Data":"5264bed43ef0b66623dec4796da7b3bb76bfbb97bffd3026398a3dccb6a16e40"} Oct 30 05:52:25.077445 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:25.077409 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:52:25.122653 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:25.122626 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:52:25.125832 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:25.125816 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:52:25.150121 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:25.150097 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:52:25.459839 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:25.459769 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:25.485499 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:25.485455 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:25.712948 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:25.712877 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:27.407014 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.406982 2578 apiserver.go:52] "Watching apiserver" Oct 30 05:52:27.430468 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.430437 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 05:52:27.446360 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.446332 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:52:27.446719 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.446696 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-wmrrw","openshift-ovn-kubernetes/ovnkube-node-zzk85","kube-system/konnectivity-agent-cq5xz","kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal","openshift-multus/multus-additional-cni-plugins-chl67","openshift-network-diagnostics/network-check-target-xsrdq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz","openshift-cluster-node-tuning-operator/tuned-85kft","openshift-multus/network-metrics-daemon-r5rsl","kube-system/global-pull-secret-syncer-bnv8q","openshift-dns/node-resolver-dz874","openshift-image-registry/node-ca-l652g","openshift-multus/multus-86c7b"] Oct 30 05:52:27.448359 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.448332 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:27.448490 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:27.448415 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:27.449520 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.449501 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:52:27.450803 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.450783 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.453453 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.453436 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dz874" Oct 30 05:52:27.454756 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.454537 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:27.454756 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:27.454601 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:27.455931 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.455912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.457389 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.457368 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.459049 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.459032 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l652g" Oct 30 05:52:27.459992 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.459976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.461814 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.461791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:27.461902 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:27.461865 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:27.462739 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.462716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.464483 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.464463 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:27.467709 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.467665 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:52:27.467709 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.467665 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:52:27.467875 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.467741 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mkzdc\"" Oct 30 05:52:27.467875 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.467822 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:52:27.467875 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.467831 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:52:27.467875 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.467845 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:52:27.468070 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.467900 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vzk7w\"" Oct 30 05:52:27.468070 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.467926 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:52:27.468070 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.467939 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:52:27.468070 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.467949 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:52:27.468070 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.468000 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mpzcb\"" Oct 30 05:52:27.468802 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.468784 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:52:27.471767 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.471742 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:52:27.471863 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.471767 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:52:27.471961 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.471945 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:52:27.472028 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.472007 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q46lp\"" Oct 30 05:52:27.472186 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.472166 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-27cwd\"" Oct 30 05:52:27.474034 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.473465 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q2ncw\"" Oct 30 05:52:27.474034 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.473596 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:52:27.474034 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.473716 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:52:27.474034 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.473814 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:52:27.474034 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.473853 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:52:27.476930 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.476907 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:52:27.477033 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.476981 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7bntx" Oct 30 05:52:27.477033 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.477026 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5hkvn\"" Oct 30 05:52:27.486300 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.486283 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7bntx" Oct 30 05:52:27.533151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.533130 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:52:27.543416 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xjq\" (UniqueName: \"kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq\") pod \"network-check-target-xsrdq\" (UID: \"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8\") " pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:27.543530 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-lib-modules\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.543530 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-host\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.543530 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cnibin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.543688 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-etc-kubernetes\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.543688 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.543688 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543633 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxjtk\" (UniqueName: \"kubernetes.io/projected/f7ce5e27-0a29-46db-a45d-94dd98cbe059-kube-api-access-cxjtk\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:52:27.543827 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysconfig\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.543827 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-multus\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.543827 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543780 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-daemon-config\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.544042 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-system-cni-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.544042 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:27.544042 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z9x5\" (UniqueName: \"kubernetes.io/projected/8a5d5930-3046-4003-bce1-4716ab70ef86-kube-api-access-9z9x5\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:52:27.544042 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.543968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-node-log\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.544042 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544023 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-conf\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.544238 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28fpx\" (UniqueName: \"kubernetes.io/projected/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-kube-api-access-28fpx\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.544238 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-kubelet\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.544238 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-var-lib-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.544238 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovn-node-metrics-cert\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.544238 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-hostroot\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.544497 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-kubernetes\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.544497 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-run\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.544497 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-log-socket\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.544497 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-bin\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.544497 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.544497 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-sys-fs\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.544497 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-socket-dir-parent\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.544794 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.544794 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544547 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-dbus\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:27.544794 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b6113ea-d183-4a72-a574-80c03c8f5352-konnectivity-ca\") pod \"konnectivity-agent-cq5xz\" (UID: \"3b6113ea-d183-4a72-a574-80c03c8f5352\") " pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:27.544794 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7212e5ac-eea5-438a-a162-74821cf38b76-hosts-file\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:52:27.544794 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-kubelet\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.544794 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-conf-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.544794 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544696 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzp62\" (UniqueName: \"kubernetes.io/projected/de6988ac-1b54-4229-9424-aa8e28cbb6c6-kube-api-access-nzp62\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.544794 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.544794 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-tuned\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-socket-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-system-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544906 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ce5e27-0a29-46db-a45d-94dd98cbe059-host\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prjhg\" (UniqueName: \"kubernetes.io/projected/cad7f8eb-8604-4b8b-a921-38b9d5c281db-kube-api-access-prjhg\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.544968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-systemd-units\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545006 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rznb5\" (UniqueName: \"kubernetes.io/projected/f1d67bea-af2e-44ad-ab9b-01123c2889fc-kube-api-access-rznb5\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-device-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-netns\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7212e5ac-eea5-438a-a162-74821cf38b76-tmp-dir\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-netd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.545141 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545137 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b6113ea-d183-4a72-a574-80c03c8f5352-agent-certs\") pod \"konnectivity-agent-cq5xz\" (UID: \"3b6113ea-d183-4a72-a574-80c03c8f5352\") " pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:27.545668 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.545668 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-tmp\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.545668 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-script-lib\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.545668 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-kubelet-config\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:27.545668 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a5d5930-3046-4003-bce1-4716ab70ef86-host-slash\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:52:27.545668 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-systemd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.545668 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545340 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.545668 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-var-lib-kubelet\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.545668 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545648 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-systemd\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.546009 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545689 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-sys\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.546009 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-env-overrides\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.546009 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhkl4\" (UniqueName: \"kubernetes.io/projected/7212e5ac-eea5-438a-a162-74821cf38b76-kube-api-access-mhkl4\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:52:27.546009 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-slash\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.546009 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cni-binary-copy\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.546009 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-multus-certs\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.546009 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx5hv\" (UniqueName: \"kubernetes.io/projected/11393f29-8fd7-4a36-b3df-03f0992fc7ad-kube-api-access-wx5hv\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.546009 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545974 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.546009 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.545993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7ce5e27-0a29-46db-a45d-94dd98cbe059-serviceca\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a5d5930-3046-4003-bce1-4716ab70ef86-iptables-alerter-script\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-modprobe-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-netns\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-config\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-k8s-cni-cncf-io\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cnibin\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-ovn\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-etc-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-registration-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mdj\" (UniqueName: \"kubernetes.io/projected/b5e82585-9fee-4f52-a796-ef82b220e6f6-kube-api-access-95mdj\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.546316 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.546282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-bin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.566810 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.566789 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:52:27.566810 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.566792 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:52:27.566975 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.566798 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:52:27.566975 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.566895 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:52:27.567089 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.567063 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:52:27.567202 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.567065 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:52:27.567202 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.567107 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:52:27.567202 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.567186 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:52:27.567202 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.567186 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:52:27.567466 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.567346 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:52:27.567466 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.567406 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d9dv5\"" Oct 30 05:52:27.567466 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.567412 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-plsgv\"" Oct 30 05:52:27.567596 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.567503 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:52:27.646767 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.646732 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-lib-modules\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.646767 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.646770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-host\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.646995 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.646796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cnibin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.646995 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.646823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-etc-kubernetes\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.646995 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.646853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.646995 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.646877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxjtk\" (UniqueName: \"kubernetes.io/projected/f7ce5e27-0a29-46db-a45d-94dd98cbe059-kube-api-access-cxjtk\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:52:27.646995 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.646901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysconfig\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.646995 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.646924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-multus\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.646995 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.646949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-daemon-config\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.646995 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.646975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-system-cni-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.646998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z9x5\" (UniqueName: \"kubernetes.io/projected/8a5d5930-3046-4003-bce1-4716ab70ef86-kube-api-access-9z9x5\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-node-log\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-conf\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28fpx\" (UniqueName: \"kubernetes.io/projected/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-kube-api-access-28fpx\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-kubelet\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-var-lib-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovn-node-metrics-cert\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-hostroot\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-kubernetes\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-run\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-log-socket\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.647298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-bin\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.648419 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.648467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-sys-fs\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.648467 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-socket-dir-parent\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-dbus\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b6113ea-d183-4a72-a574-80c03c8f5352-konnectivity-ca\") pod \"konnectivity-agent-cq5xz\" (UID: \"3b6113ea-d183-4a72-a574-80c03c8f5352\") " pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647792 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cnibin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647706 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-lib-modules\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647822 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-etc-kubernetes\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647872 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-var-lib-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647952 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648129 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysconfig\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648136 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647850 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-kubelet\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648177 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-system-cni-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.648566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-hostroot\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648566 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-socket-dir-parent\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647557 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-node-log\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648323 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-run\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648350 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-kubernetes\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648372 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-log-socket\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647745 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-host\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-bin\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7212e5ac-eea5-438a-a162-74821cf38b76-hosts-file\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648152 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-multus\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647519 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648602 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-sys-fs\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.647657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-conf\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648565 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-daemon-config\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648687 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-kubelet\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:27.648227 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-kubelet\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-conf-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.649187 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649068 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzp62\" (UniqueName: \"kubernetes.io/projected/de6988ac-1b54-4229-9424-aa8e28cbb6c6-kube-api-access-nzp62\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-conf-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.648671 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7212e5ac-eea5-438a-a162-74821cf38b76-hosts-file\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649146 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-dbus\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:27.649156 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret podName:d44df296-4186-4c40-81de-1ed06c15955e nodeName:}" failed. No retries permitted until 2026-10-30 05:52:28.149137932 +0000 UTC m=+3.058871130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret") pod "global-pull-secret-syncer-bnv8q" (UID: "d44df296-4186-4c40-81de-1ed06c15955e") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649156 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b6113ea-d183-4a72-a574-80c03c8f5352-konnectivity-ca\") pod \"konnectivity-agent-cq5xz\" (UID: \"3b6113ea-d183-4a72-a574-80c03c8f5352\") " pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-tuned\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-socket-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649214 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-system-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649249 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ce5e27-0a29-46db-a45d-94dd98cbe059-host\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prjhg\" (UniqueName: \"kubernetes.io/projected/cad7f8eb-8604-4b8b-a921-38b9d5c281db-kube-api-access-prjhg\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-systemd-units\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649330 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-system-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.650017 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649329 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649363 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ce5e27-0a29-46db-a45d-94dd98cbe059-host\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649376 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-socket-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rznb5\" (UniqueName: \"kubernetes.io/projected/f1d67bea-af2e-44ad-ab9b-01123c2889fc-kube-api-access-rznb5\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649403 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-systemd-units\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-device-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649431 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-netns\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7212e5ac-eea5-438a-a162-74821cf38b76-tmp-dir\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649480 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-device-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-netd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649519 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-netns\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b6113ea-d183-4a72-a574-80c03c8f5352-agent-certs\") pod \"konnectivity-agent-cq5xz\" (UID: \"3b6113ea-d183-4a72-a574-80c03c8f5352\") " pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649562 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-netd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649572 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.650774 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-tmp\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-script-lib\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-kubelet-config\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a5d5930-3046-4003-bce1-4716ab70ef86-host-slash\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649695 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-kubelet-config\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-systemd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649736 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7212e5ac-eea5-438a-a162-74821cf38b76-tmp-dir\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-var-lib-kubelet\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-systemd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-systemd\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-sys\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649827 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a5d5930-3046-4003-bce1-4716ab70ef86-host-slash\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649864 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649876 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-sys\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649905 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-var-lib-kubelet\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-env-overrides\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.651592 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhkl4\" (UniqueName: \"kubernetes.io/projected/7212e5ac-eea5-438a-a162-74821cf38b76-kube-api-access-mhkl4\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649947 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-systemd\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-slash\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.649982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cni-binary-copy\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-multus-certs\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650053 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-slash\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx5hv\" (UniqueName: \"kubernetes.io/projected/11393f29-8fd7-4a36-b3df-03f0992fc7ad-kube-api-access-wx5hv\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7ce5e27-0a29-46db-a45d-94dd98cbe059-serviceca\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a5d5930-3046-4003-bce1-4716ab70ef86-iptables-alerter-script\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-modprobe-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-netns\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-config\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-k8s-cni-cncf-io\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cnibin\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-ovn\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-etc-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.652355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-registration-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mdj\" (UniqueName: \"kubernetes.io/projected/b5e82585-9fee-4f52-a796-ef82b220e6f6-kube-api-access-95mdj\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-bin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xjq\" (UniqueName: \"kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq\") pod \"network-check-target-xsrdq\" (UID: \"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8\") " pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.650803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-multus-certs\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651078 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cnibin\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-env-overrides\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651498 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:27.651516 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651561 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-bin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:27.651592 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:52:28.151573706 +0000 UTC m=+3.061306932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651611 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-netns\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651638 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-k8s-cni-cncf-io\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-ovn\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.653119 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651711 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.653824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651732 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-etc-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.653824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-modprobe-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.653824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651587 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7ce5e27-0a29-46db-a45d-94dd98cbe059-serviceca\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:52:27.653824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.651829 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-registration-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.653824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.652095 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-config\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.653824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.652150 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a5d5930-3046-4003-bce1-4716ab70ef86-iptables-alerter-script\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:52:27.653824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.652229 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cni-binary-copy\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.653824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.652646 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-script-lib\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.653824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.653274 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovn-node-metrics-cert\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.653824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.653360 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-tuned\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.655509 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.655487 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b6113ea-d183-4a72-a574-80c03c8f5352-agent-certs\") pod \"konnectivity-agent-cq5xz\" (UID: \"3b6113ea-d183-4a72-a574-80c03c8f5352\") " pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:27.655610 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.655595 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-tmp\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.670591 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.670529 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxjtk\" (UniqueName: \"kubernetes.io/projected/f7ce5e27-0a29-46db-a45d-94dd98cbe059-kube-api-access-cxjtk\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:52:27.672270 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:27.672250 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:27.672270 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:27.672272 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:27.672453 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:27.672286 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d2xjq for pod openshift-network-diagnostics/network-check-target-xsrdq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:27.672453 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:27.672365 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq podName:4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:28.172347785 +0000 UTC m=+3.082080999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d2xjq" (UniqueName: "kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq") pod "network-check-target-xsrdq" (UID: "4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:27.672553 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.672527 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z9x5\" (UniqueName: \"kubernetes.io/projected/8a5d5930-3046-4003-bce1-4716ab70ef86-kube-api-access-9z9x5\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:52:27.673131 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.673107 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28fpx\" (UniqueName: \"kubernetes.io/projected/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-kube-api-access-28fpx\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.677171 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.677138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhkl4\" (UniqueName: \"kubernetes.io/projected/7212e5ac-eea5-438a-a162-74821cf38b76-kube-api-access-mhkl4\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:52:27.677269 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.677224 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prjhg\" (UniqueName: \"kubernetes.io/projected/cad7f8eb-8604-4b8b-a921-38b9d5c281db-kube-api-access-prjhg\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:27.677269 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.677227 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mdj\" (UniqueName: \"kubernetes.io/projected/b5e82585-9fee-4f52-a796-ef82b220e6f6-kube-api-access-95mdj\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.678996 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.678755 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzp62\" (UniqueName: \"kubernetes.io/projected/de6988ac-1b54-4229-9424-aa8e28cbb6c6-kube-api-access-nzp62\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.678996 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.678953 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rznb5\" (UniqueName: \"kubernetes.io/projected/f1d67bea-af2e-44ad-ab9b-01123c2889fc-kube-api-access-rznb5\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.680482 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.680464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx5hv\" (UniqueName: \"kubernetes.io/projected/11393f29-8fd7-4a36-b3df-03f0992fc7ad-kube-api-access-wx5hv\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.762763 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.762728 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:52:27.771676 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.771647 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86c7b" Oct 30 05:52:27.773775 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:27.773746 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5d5930_3046_4003_bce1_4716ab70ef86.slice/crio-98a66792bd4248f2fdec8187af68dddc5fd7e1057d997408076f5b40c902e33c WatchSource:0}: Error finding container 98a66792bd4248f2fdec8187af68dddc5fd7e1057d997408076f5b40c902e33c: Status 404 returned error can't find the container with id 98a66792bd4248f2fdec8187af68dddc5fd7e1057d997408076f5b40c902e33c Oct 30 05:52:27.781709 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.781689 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dz874" Oct 30 05:52:27.782442 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:27.782416 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11393f29_8fd7_4a36_b3df_03f0992fc7ad.slice/crio-5a56192fcd20a9af9aad204e5d2162d757e993b589a9e3908b4188cb024d17e8 WatchSource:0}: Error finding container 5a56192fcd20a9af9aad204e5d2162d757e993b589a9e3908b4188cb024d17e8: Status 404 returned error can't find the container with id 5a56192fcd20a9af9aad204e5d2162d757e993b589a9e3908b4188cb024d17e8 Oct 30 05:52:27.795114 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.795091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:52:27.796181 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.796161 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:52:27.797775 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:27.797749 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7212e5ac_eea5_438a_a162_74821cf38b76.slice/crio-a7c4fcdeb47e184e4c5305e69352178adb81dc48402fd7b3a6fb694cd0ee0697 WatchSource:0}: Error finding container a7c4fcdeb47e184e4c5305e69352178adb81dc48402fd7b3a6fb694cd0ee0697: Status 404 returned error can't find the container with id a7c4fcdeb47e184e4c5305e69352178adb81dc48402fd7b3a6fb694cd0ee0697 Oct 30 05:52:27.803813 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.803787 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l652g" Oct 30 05:52:27.804766 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:27.804737 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6988ac_1b54_4229_9424_aa8e28cbb6c6.slice/crio-0e40d083151df3cf253eb758bac5c948afc8977d7cacbd1beb526d8258094ee1 WatchSource:0}: Error finding container 0e40d083151df3cf253eb758bac5c948afc8977d7cacbd1beb526d8258094ee1: Status 404 returned error can't find the container with id 0e40d083151df3cf253eb758bac5c948afc8977d7cacbd1beb526d8258094ee1 Oct 30 05:52:27.812982 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:27.812848 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ce5e27_0a29_46db_a45d_94dd98cbe059.slice/crio-9a10d6ef400c2aaada873e76b1aba180329a6c526c265c296884109e30e9dbbe WatchSource:0}: Error finding container 9a10d6ef400c2aaada873e76b1aba180329a6c526c265c296884109e30e9dbbe: Status 404 returned error can't find the container with id 9a10d6ef400c2aaada873e76b1aba180329a6c526c265c296884109e30e9dbbe Oct 30 05:52:27.816477 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.816462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:27.822033 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.822011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:52:27.823123 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:27.823105 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:27.847432 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:27.847409 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6113ea_d183_4a72_a574_80c03c8f5352.slice/crio-5cc4e16f243d0dbbf7104b96dc1cfb68747d1ea324b3f55e870ac53b7664d209 WatchSource:0}: Error finding container 5cc4e16f243d0dbbf7104b96dc1cfb68747d1ea324b3f55e870ac53b7664d209: Status 404 returned error can't find the container with id 5cc4e16f243d0dbbf7104b96dc1cfb68747d1ea324b3f55e870ac53b7664d209 Oct 30 05:52:27.848791 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:27.848658 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d67bea_af2e_44ad_ab9b_01123c2889fc.slice/crio-f941e4e4b67f967090a4170d32b43b293f91cf56a16c0bf5af4ec29cb49c2613 WatchSource:0}: Error finding container f941e4e4b67f967090a4170d32b43b293f91cf56a16c0bf5af4ec29cb49c2613: Status 404 returned error can't find the container with id f941e4e4b67f967090a4170d32b43b293f91cf56a16c0bf5af4ec29cb49c2613 Oct 30 05:52:28.154755 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.154721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:28.154905 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.154785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:28.154960 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:28.154918 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:28.155004 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:28.154979 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret podName:d44df296-4186-4c40-81de-1ed06c15955e nodeName:}" failed. No retries permitted until 2026-10-30 05:52:29.1549606 +0000 UTC m=+4.064693816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret") pod "global-pull-secret-syncer-bnv8q" (UID: "d44df296-4186-4c40-81de-1ed06c15955e") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:28.155109 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:28.155084 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:28.155237 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:28.155149 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:52:29.155135171 +0000 UTC m=+4.064868384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:28.256883 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.256120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xjq\" (UniqueName: \"kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq\") pod \"network-check-target-xsrdq\" (UID: \"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8\") " pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:28.256883 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:28.256369 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:28.256883 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:28.256392 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:28.256883 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:28.256406 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d2xjq for pod openshift-network-diagnostics/network-check-target-xsrdq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:28.256883 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:28.256464 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq podName:4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:29.25644765 +0000 UTC m=+4.166180862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d2xjq" (UniqueName: "kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq") pod "network-check-target-xsrdq" (UID: "4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:28.490359 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.489965 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:47:27 +0000 UTC" deadline="2028-04-16 07:37:23.030944972 +0000 UTC" Oct 30 05:52:28.490359 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.490244 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12817h44m54.540707291s" Oct 30 05:52:28.589911 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.589871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:28.590076 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:28.590028 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:28.611865 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.611829 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wmrrw" event={"ID":"8a5d5930-3046-4003-bce1-4716ab70ef86","Type":"ContainerStarted","Data":"98a66792bd4248f2fdec8187af68dddc5fd7e1057d997408076f5b40c902e33c"} Oct 30 05:52:28.632755 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.632717 2578 generic.go:356] "Generic (PLEG): container finished" podID="1bb6269644393a710c6775a446a340ab" containerID="62c14a981bdb43f4e95d9f612272ce4648e5c816224421cb627efe5de9874865" exitCode=0 Oct 30 05:52:28.632923 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.632811 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerDied","Data":"62c14a981bdb43f4e95d9f612272ce4648e5c816224421cb627efe5de9874865"} Oct 30 05:52:28.641728 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.641601 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" event={"ID":"f1d67bea-af2e-44ad-ab9b-01123c2889fc","Type":"ContainerStarted","Data":"f941e4e4b67f967090a4170d32b43b293f91cf56a16c0bf5af4ec29cb49c2613"} Oct 30 05:52:28.649335 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.649289 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l652g" event={"ID":"f7ce5e27-0a29-46db-a45d-94dd98cbe059","Type":"ContainerStarted","Data":"9a10d6ef400c2aaada873e76b1aba180329a6c526c265c296884109e30e9dbbe"} Oct 30 05:52:28.675567 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.660529 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-85kft" event={"ID":"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d","Type":"ContainerStarted","Data":"a0a6cbff63a045a68a5e583993a0c01a3412975c963ec4853890a20afa8ee060"} Oct 30 05:52:28.676154 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.676126 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" event={"ID":"ed322e80a89b2bbeafbf0f88fe9597d6","Type":"ContainerStarted","Data":"9f8bb74f3dff7f361bf929c7fbe58244c6deac571caa8e58d17ced38e59f8533"} Oct 30 05:52:28.679229 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.679198 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" event={"ID":"b5e82585-9fee-4f52-a796-ef82b220e6f6","Type":"ContainerStarted","Data":"d0fd2dfc7fa02952cd90a71420c4870c6f0e1fc33234dec4eac740c3c937504a"} Oct 30 05:52:28.691492 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.691464 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cq5xz" event={"ID":"3b6113ea-d183-4a72-a574-80c03c8f5352","Type":"ContainerStarted","Data":"5cc4e16f243d0dbbf7104b96dc1cfb68747d1ea324b3f55e870ac53b7664d209"} Oct 30 05:52:28.697926 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.697895 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chl67" event={"ID":"de6988ac-1b54-4229-9424-aa8e28cbb6c6","Type":"ContainerStarted","Data":"0e40d083151df3cf253eb758bac5c948afc8977d7cacbd1beb526d8258094ee1"} Oct 30 05:52:28.701920 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.701890 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dz874" event={"ID":"7212e5ac-eea5-438a-a162-74821cf38b76","Type":"ContainerStarted","Data":"a7c4fcdeb47e184e4c5305e69352178adb81dc48402fd7b3a6fb694cd0ee0697"} Oct 30 05:52:28.717802 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:28.717775 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86c7b" event={"ID":"11393f29-8fd7-4a36-b3df-03f0992fc7ad","Type":"ContainerStarted","Data":"5a56192fcd20a9af9aad204e5d2162d757e993b589a9e3908b4188cb024d17e8"} Oct 30 05:52:29.165906 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:29.165801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:29.165906 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:29.165883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:29.166108 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:29.166004 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:29.166108 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:29.166062 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:52:31.166045207 +0000 UTC m=+6.075778422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:29.166484 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:29.166465 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:29.166574 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:29.166518 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret podName:d44df296-4186-4c40-81de-1ed06c15955e nodeName:}" failed. No retries permitted until 2026-10-30 05:52:31.166503797 +0000 UTC m=+6.076237011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret") pod "global-pull-secret-syncer-bnv8q" (UID: "d44df296-4186-4c40-81de-1ed06c15955e") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:29.266644 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:29.266594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xjq\" (UniqueName: \"kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq\") pod \"network-check-target-xsrdq\" (UID: \"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8\") " pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:29.266817 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:29.266763 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:29.266817 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:29.266781 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:29.266817 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:29.266792 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d2xjq for pod openshift-network-diagnostics/network-check-target-xsrdq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:29.267019 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:29.266845 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq podName:4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:31.266828946 +0000 UTC m=+6.176562164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d2xjq" (UniqueName: "kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq") pod "network-check-target-xsrdq" (UID: "4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:29.491118 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:29.491078 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:47:27 +0000 UTC" deadline="2028-04-30 09:44:10.49739932 +0000 UTC" Oct 30 05:52:29.491118 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:29.491118 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13155h51m41.006286164s" Oct 30 05:52:29.592546 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:29.592512 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:29.592721 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:29.592649 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:29.593085 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:29.593059 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:29.593173 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:29.593154 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:29.747520 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:29.746827 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerStarted","Data":"9844588068de4d8f2b7e4add7e0eb38e65d73c7fac039792c734686970ea2298"} Oct 30 05:52:29.789444 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:29.789382 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" podStartSLOduration=3.789361049 podStartE2EDuration="3.789361049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:28.714199671 +0000 UTC m=+3.623932893" watchObservedRunningTime="2026-10-30 05:52:29.789361049 +0000 UTC m=+4.699094271" Oct 30 05:52:29.789632 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:29.789489 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" podStartSLOduration=3.7894796570000002 podStartE2EDuration="3.789479657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:29.788779975 +0000 UTC m=+4.698513198" watchObservedRunningTime="2026-10-30 05:52:29.789479657 +0000 UTC m=+4.699212880" Oct 30 05:52:30.590237 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:30.590203 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:30.590742 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:30.590370 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:30.752051 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:30.751967 2578 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="cccafdb09c457277de081157c9eeb3cf7c955083151fda3fa533313f24236185" exitCode=0 Oct 30 05:52:30.752659 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:30.752627 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chl67" event={"ID":"de6988ac-1b54-4229-9424-aa8e28cbb6c6","Type":"ContainerDied","Data":"cccafdb09c457277de081157c9eeb3cf7c955083151fda3fa533313f24236185"} Oct 30 05:52:31.186427 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:31.186334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:31.186427 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:31.186406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:31.186633 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:31.186618 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:31.186696 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:31.186686 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret podName:d44df296-4186-4c40-81de-1ed06c15955e nodeName:}" failed. No retries permitted until 2026-10-30 05:52:35.186668548 +0000 UTC m=+10.096401753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret") pod "global-pull-secret-syncer-bnv8q" (UID: "d44df296-4186-4c40-81de-1ed06c15955e") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:31.186763 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:31.186754 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:31.186812 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:31.186786 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:52:35.186775229 +0000 UTC m=+10.096508433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:31.287668 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:31.287005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xjq\" (UniqueName: \"kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq\") pod \"network-check-target-xsrdq\" (UID: \"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8\") " pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:31.287668 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:31.287247 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:31.287668 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:31.287270 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:31.287668 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:31.287282 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d2xjq for pod openshift-network-diagnostics/network-check-target-xsrdq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:31.287668 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:31.287357 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq podName:4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:35.287340376 +0000 UTC m=+10.197073591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d2xjq" (UniqueName: "kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq") pod "network-check-target-xsrdq" (UID: "4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:31.592978 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:31.592434 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:31.592978 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:31.592565 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:31.592978 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:31.592617 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:31.592978 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:31.592736 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:31.756276 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:31.756189 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cq5xz" event={"ID":"3b6113ea-d183-4a72-a574-80c03c8f5352","Type":"ContainerStarted","Data":"5ce71be437829ff7e43dd169c116d794cd00ddfe67836e2aedcf4e06e1e465d7"} Oct 30 05:52:31.885066 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:31.885033 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:31.885557 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:31.885536 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:31.925191 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:31.925130 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cq5xz" podStartSLOduration=2.369484119 podStartE2EDuration="5.925109518s" totalImagesPullingTime="3.555625399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:31.798454427 +0000 UTC m=+6.708187650" watchObservedRunningTime="2026-10-30 05:52:31.925109518 +0000 UTC m=+6.834842741" Oct 30 05:52:32.589939 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:32.589420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:32.589939 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:32.589554 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:32.763601 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:32.763373 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l652g" event={"ID":"f7ce5e27-0a29-46db-a45d-94dd98cbe059","Type":"ContainerStarted","Data":"15e029e9d90fbc1896b221df5b06be9d570f6f71abdaab09fa99c29a7d1864b5"} Oct 30 05:52:33.590576 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:33.589911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:33.590576 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:33.590045 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:33.590576 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:33.590441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:33.590576 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:33.590536 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:33.769749 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:33.769671 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" event={"ID":"b5e82585-9fee-4f52-a796-ef82b220e6f6","Type":"ContainerStarted","Data":"1cb9864e0d47d1ba5275f37d66a6d86060fb4824635c85df89e762e8423c9004"} Oct 30 05:52:33.770208 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:33.769771 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:52:34.589992 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:34.589916 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:34.590149 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:34.590076 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:35.220986 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.220234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:35.220986 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.220322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:35.220986 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:35.220453 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:35.220986 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:35.220513 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret podName:d44df296-4186-4c40-81de-1ed06c15955e nodeName:}" failed. No retries permitted until 2026-10-30 05:52:43.220496553 +0000 UTC m=+18.130229781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret") pod "global-pull-secret-syncer-bnv8q" (UID: "d44df296-4186-4c40-81de-1ed06c15955e") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:35.220986 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:35.220898 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:35.220986 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:35.220950 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:52:43.220934601 +0000 UTC m=+18.130667820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:35.321145 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.321108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xjq\" (UniqueName: \"kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq\") pod \"network-check-target-xsrdq\" (UID: \"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8\") " pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:35.321339 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:35.321300 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:35.321429 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:35.321347 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:35.321429 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:35.321360 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d2xjq for pod openshift-network-diagnostics/network-check-target-xsrdq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:35.321429 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:35.321426 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq podName:4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:43.321406521 +0000 UTC m=+18.231139737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d2xjq" (UniqueName: "kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq") pod "network-check-target-xsrdq" (UID: "4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:35.537163 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.536854 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:52:35.592384 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.590532 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:35.592384 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:35.590607 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:35.592384 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.590726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:35.592384 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:35.590774 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:35.776729 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.776644 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" event={"ID":"b5e82585-9fee-4f52-a796-ef82b220e6f6","Type":"ContainerStarted","Data":"1f1f61355a6e7b820f613e7c2f702893fea1aa675324a1793024b366a52c566c"} Oct 30 05:52:35.778867 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.778834 2578 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="ecbd10b7176d719993bc30656c927c4174abd95b3977df172dd25784ac3f94ec" exitCode=0 Oct 30 05:52:35.778992 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.778925 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chl67" event={"ID":"de6988ac-1b54-4229-9424-aa8e28cbb6c6","Type":"ContainerDied","Data":"ecbd10b7176d719993bc30656c927c4174abd95b3977df172dd25784ac3f94ec"} Oct 30 05:52:35.780823 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.780483 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dz874" event={"ID":"7212e5ac-eea5-438a-a162-74821cf38b76","Type":"ContainerStarted","Data":"dcb6573917eae81ab0651cd96247d2ddf6e8c89fbed134016fc3ae314b039c78"} Oct 30 05:52:35.782039 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.782019 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86c7b" event={"ID":"11393f29-8fd7-4a36-b3df-03f0992fc7ad","Type":"ContainerStarted","Data":"231747be5a8b5bd9136221543963204b46cb23dc934aa5f883d03a8b0ee2077d"} Oct 30 05:52:35.849900 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.849845 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l652g" podStartSLOduration=6.239417825 podStartE2EDuration="10.84982635s" totalImagesPullingTime="4.610408525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:32.802241669 +0000 UTC m=+7.711974901" watchObservedRunningTime="2026-10-30 05:52:35.84982635 +0000 UTC m=+10.759559574" Oct 30 05:52:35.894015 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:35.893958 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-86c7b" podStartSLOduration=3.497621552 podStartE2EDuration="10.893945047s" totalImagesPullingTime="7.396323495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:35.893910421 +0000 UTC m=+10.803643642" watchObservedRunningTime="2026-10-30 05:52:35.893945047 +0000 UTC m=+10.803678267" Oct 30 05:52:36.494508 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:36.493739 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:52:35.536872795Z","UUID":"dc65b9c3-6601-4fcc-a6e5-4685f893a748","Handler":null,"Name":"","Endpoint":""} Oct 30 05:52:36.496511 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:36.496032 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:52:36.496511 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:36.496080 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:52:36.590097 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:36.590072 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:36.590229 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:36.590202 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:36.787168 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:36.787135 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" event={"ID":"b5e82585-9fee-4f52-a796-ef82b220e6f6","Type":"ContainerStarted","Data":"e793120da2c5c370fe7d52091d4d4b9c17d57d92ddf56d9ea00de1deba651c10"} Oct 30 05:52:36.789756 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:36.789726 2578 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="bfaf8bf4d86fbe25bfb02dfab0fce1c82d299176a46fc72b048fff24f4fe17a0" exitCode=0 Oct 30 05:52:36.789901 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:36.789820 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chl67" event={"ID":"de6988ac-1b54-4229-9424-aa8e28cbb6c6","Type":"ContainerDied","Data":"bfaf8bf4d86fbe25bfb02dfab0fce1c82d299176a46fc72b048fff24f4fe17a0"} Oct 30 05:52:36.831410 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:36.831356 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" podStartSLOduration=2.624132354 podStartE2EDuration="10.831340966s" totalImagesPullingTime="8.207208612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:36.831057645 +0000 UTC m=+11.740790871" watchObservedRunningTime="2026-10-30 05:52:36.831340966 +0000 UTC m=+11.741074187" Oct 30 05:52:36.831658 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:36.831626 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dz874" podStartSLOduration=4.212576512 podStartE2EDuration="11.831614899s" totalImagesPullingTime="7.619038387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:35.932675164 +0000 UTC m=+10.842408386" watchObservedRunningTime="2026-10-30 05:52:36.831614899 +0000 UTC m=+11.741348119" Oct 30 05:52:37.589957 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:37.589717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:37.590400 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:37.589717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:37.590400 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:37.590081 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:37.590400 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:37.590168 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:37.798455 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:37.798424 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wmrrw" event={"ID":"8a5d5930-3046-4003-bce1-4716ab70ef86","Type":"ContainerStarted","Data":"0ef8b2c0dac812813f6937d0964581a06ecbd02b1835d1d0e2171aeb6b2b2d04"} Oct 30 05:52:37.838336 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:37.838260 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wmrrw" podStartSLOduration=5.203569096 podStartE2EDuration="12.838238748s" totalImagesPullingTime="7.634669652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:37.837184465 +0000 UTC m=+12.746917687" watchObservedRunningTime="2026-10-30 05:52:37.838238748 +0000 UTC m=+12.747971971" Oct 30 05:52:38.589375 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:38.589338 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:38.589573 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:38.589470 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:38.802251 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:38.802220 2578 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="eda7ff193b1121c44447cdc508f8cc42be5dba763a2abb45e1160457608d3734" exitCode=0 Oct 30 05:52:38.802945 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:38.802325 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chl67" event={"ID":"de6988ac-1b54-4229-9424-aa8e28cbb6c6","Type":"ContainerDied","Data":"eda7ff193b1121c44447cdc508f8cc42be5dba763a2abb45e1160457608d3734"} Oct 30 05:52:38.803912 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:38.803883 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-85kft" event={"ID":"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d","Type":"ContainerStarted","Data":"720349a36559cb40bbb3ab13ca01ca54af6ff17b718e159a03c72f1e534989a4"} Oct 30 05:52:38.920893 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:38.920800 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-85kft" podStartSLOduration=3.954663322 podStartE2EDuration="13.920784289s" totalImagesPullingTime="9.966120967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:38.92070681 +0000 UTC m=+13.830440028" watchObservedRunningTime="2026-10-30 05:52:38.920784289 +0000 UTC m=+13.830517512" Oct 30 05:52:39.589523 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:39.589484 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:39.589703 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:39.589613 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:39.589982 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:39.589959 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:39.590092 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:39.590037 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:40.590078 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:40.590055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:40.590623 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:40.590193 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:40.811128 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:40.810878 2578 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="9d308222186dd9a3ba6359b4412dab6472f7841c46ebd64fa4a590ed897079ea" exitCode=0 Oct 30 05:52:40.811128 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:40.810952 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chl67" event={"ID":"de6988ac-1b54-4229-9424-aa8e28cbb6c6","Type":"ContainerDied","Data":"9d308222186dd9a3ba6359b4412dab6472f7841c46ebd64fa4a590ed897079ea"} Oct 30 05:52:41.592274 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:41.592244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:41.592741 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:41.592384 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:41.592741 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:41.592430 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:41.592741 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:41.592509 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:41.815982 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:41.815947 2578 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="6d13c54a0a19a721dad4c7218b2e8dec4dc6896edca9fc8de7abe899762dbf03" exitCode=0 Oct 30 05:52:41.816147 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:41.816001 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chl67" event={"ID":"de6988ac-1b54-4229-9424-aa8e28cbb6c6","Type":"ContainerDied","Data":"6d13c54a0a19a721dad4c7218b2e8dec4dc6896edca9fc8de7abe899762dbf03"} Oct 30 05:52:42.590067 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:42.590045 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:42.590157 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:42.590144 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:42.823738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:42.823495 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" event={"ID":"f1d67bea-af2e-44ad-ab9b-01123c2889fc","Type":"ContainerStarted","Data":"c058fc55052f73262b8c6807971fffac3c51d861e3f1c7835e493b0f5b699990"} Oct 30 05:52:42.824178 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:42.823769 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" event={"ID":"f1d67bea-af2e-44ad-ab9b-01123c2889fc","Type":"ContainerStarted","Data":"61803cd0d10bd367e69d4462a245a65dfc2060d2f4c26f1f250f5a88b9041818"} Oct 30 05:52:42.824178 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:42.823821 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" event={"ID":"f1d67bea-af2e-44ad-ab9b-01123c2889fc","Type":"ContainerStarted","Data":"bd802f5c0c89297aa497d63c523da0a8fcc87c4c171431c02b73199facb97123"} Oct 30 05:52:42.824178 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:42.823874 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" event={"ID":"f1d67bea-af2e-44ad-ab9b-01123c2889fc","Type":"ContainerStarted","Data":"4b57454f34703730dcaf642e028e131f59bd6c5e675224274d37d4a0c9aa16b7"} Oct 30 05:52:42.824178 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:42.823920 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" event={"ID":"f1d67bea-af2e-44ad-ab9b-01123c2889fc","Type":"ContainerStarted","Data":"86298eafeb82f47e2b4435696806c0b986fcbf338d0495ea96b94c6c116afdfd"} Oct 30 05:52:42.824178 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:42.823947 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" event={"ID":"f1d67bea-af2e-44ad-ab9b-01123c2889fc","Type":"ContainerStarted","Data":"18bd619620cb3b2a8eabba765954438fc59bbce0e2f02b1686f893664b873fa5"} Oct 30 05:52:42.826850 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:42.826817 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chl67" event={"ID":"de6988ac-1b54-4229-9424-aa8e28cbb6c6","Type":"ContainerStarted","Data":"f77a34e626b460f58d2683319b04c63f5f58a57c3df7f7f2533d2860ce840b7f"} Oct 30 05:52:42.875794 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:42.875752 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-chl67" podStartSLOduration=7.531136783 podStartE2EDuration="17.875741435s" totalImagesPullingTime="10.344604652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:42.875586548 +0000 UTC m=+17.785319771" watchObservedRunningTime="2026-10-30 05:52:42.875741435 +0000 UTC m=+17.785474673" Oct 30 05:52:43.281483 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:43.281399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:43.281483 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:43.281444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:43.281671 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:43.281539 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:43.281671 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:43.281540 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:43.281671 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:43.281590 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret podName:d44df296-4186-4c40-81de-1ed06c15955e nodeName:}" failed. No retries permitted until 2026-10-30 05:52:59.281577011 +0000 UTC m=+34.191310210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret") pod "global-pull-secret-syncer-bnv8q" (UID: "d44df296-4186-4c40-81de-1ed06c15955e") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:43.281671 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:43.281602 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:52:59.281595818 +0000 UTC m=+34.191329016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:43.382431 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:43.382402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xjq\" (UniqueName: \"kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq\") pod \"network-check-target-xsrdq\" (UID: \"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8\") " pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:43.382563 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:43.382549 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:43.382598 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:43.382566 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:43.382598 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:43.382575 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d2xjq for pod openshift-network-diagnostics/network-check-target-xsrdq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:43.382680 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:43.382618 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq podName:4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:59.382605808 +0000 UTC m=+34.292339008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d2xjq" (UniqueName: "kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq") pod "network-check-target-xsrdq" (UID: "4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:43.589458 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:43.589375 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:43.589617 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:43.589380 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:43.589617 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:43.589506 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:43.589617 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:43.589555 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:44.589811 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:44.589778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:44.590173 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:44.589889 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:44.833802 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:44.833762 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" event={"ID":"f1d67bea-af2e-44ad-ab9b-01123c2889fc","Type":"ContainerStarted","Data":"6083f5e7be0a347e9a3ff78c47a34e233c153a77dbc46c51beb3a673cdd9d8cb"} Oct 30 05:52:45.590556 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:45.590517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:45.591042 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:45.590783 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:45.591568 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:45.591544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:45.591718 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:45.591688 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:46.589419 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:46.589381 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:46.589568 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:46.589502 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:46.842332 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:46.842243 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" event={"ID":"f1d67bea-af2e-44ad-ab9b-01123c2889fc","Type":"ContainerStarted","Data":"3c168e2dffa01b9727d44a6f221c335b56dd84a72a3f230d45798ed0329b768a"} Oct 30 05:52:46.842945 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:46.842451 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:46.842945 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:46.842513 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:46.842945 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:46.842563 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:46.858145 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:46.858107 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:46.858362 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:46.858275 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:52:46.932180 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:46.932135 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" podStartSLOduration=7.497183589 podStartE2EDuration="21.932122529s" totalImagesPullingTime="14.43493894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:46.932016497 +0000 UTC m=+21.841749719" watchObservedRunningTime="2026-10-30 05:52:46.932122529 +0000 UTC m=+21.841855750" Oct 30 05:52:47.589511 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:47.589477 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:47.589511 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:47.589499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:47.589853 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:47.589612 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:47.589853 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:47.589716 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:48.590014 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:48.589986 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:48.590665 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:48.590128 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:48.855006 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:48.854941 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bnv8q"] Oct 30 05:52:48.855137 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:48.855030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:48.855137 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:48.855109 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:48.858808 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:48.858777 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsrdq"] Oct 30 05:52:48.858910 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:48.858870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:48.858970 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:48.858951 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:48.866755 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:48.866733 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5rsl"] Oct 30 05:52:48.866869 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:48.866827 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:48.866952 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:48.866930 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:50.589707 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:50.589666 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:50.590128 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:50.589667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:50.590128 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:50.589780 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:50.590128 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:50.589847 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:50.590128 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:50.589667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:50.590128 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:50.589912 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:52.590382 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:52.590188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:52.590755 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:52.590241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:52.590755 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:52.590456 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:52.590755 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:52.590247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:52.590755 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:52.590536 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:52.590755 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:52.590611 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:53.101805 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:53.101773 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:53.101980 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:53.101943 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:52:53.102339 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:53.102291 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:52:54.589698 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.589673 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:54.589698 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.589692 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:54.590089 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.589705 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:54.590089 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:54.589798 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5rsl" podUID="cad7f8eb-8604-4b8b-a921-38b9d5c281db" Oct 30 05:52:54.590089 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:54.589948 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsrdq" podUID="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" Oct 30 05:52:54.590089 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:54.590049 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bnv8q" podUID="d44df296-4186-4c40-81de-1ed06c15955e" Oct 30 05:52:54.861499 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.861476 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeReady" Oct 30 05:52:54.861658 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.861588 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 05:52:54.914196 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.914160 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-74dvn"] Oct 30 05:52:54.917927 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.917912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-74dvn" Oct 30 05:52:54.920885 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.920860 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j7pwr"] Oct 30 05:52:54.924234 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.924211 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2626d"] Oct 30 05:52:54.924364 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.924347 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j7pwr" Oct 30 05:52:54.927471 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.927450 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-lq9h9"] Oct 30 05:52:54.927596 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.927571 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2626d" Oct 30 05:52:54.930496 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.930478 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5b994dfc85-2v9ww"] Oct 30 05:52:54.930584 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.930570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:54.935033 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.935014 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf"] Oct 30 05:52:54.935132 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.935105 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:54.938125 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.938103 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 05:52:54.938285 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.938269 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 05:52:54.939214 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.939198 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7ztmq"] Oct 30 05:52:54.940732 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.940327 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:52:54.946504 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.946484 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b9c9cf4b4-89zvd"] Oct 30 05:52:54.946603 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.946588 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:54.949755 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.949382 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr"] Oct 30 05:52:54.952422 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.952404 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:54.954262 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.954241 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 05:52:54.954402 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.954327 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 05:52:54.954402 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.954245 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 05:52:54.954402 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.954373 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 05:52:54.954617 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.954428 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj"] Oct 30 05:52:54.954617 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.954525 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:54.957352 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.957329 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq"] Oct 30 05:52:54.957453 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.957439 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:54.960407 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.960388 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w"] Oct 30 05:52:54.960524 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.960505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:52:54.962136 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.962113 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 05:52:54.964223 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.964205 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5twq8"] Oct 30 05:52:54.964470 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.964345 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w" Oct 30 05:52:54.967534 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.967503 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z"] Oct 30 05:52:54.967926 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.967815 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:52:54.971484 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.971469 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft"] Oct 30 05:52:54.972333 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.971717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:54.975283 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.975263 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-74dvn"] Oct 30 05:52:54.975407 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.975324 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5gv9v"] Oct 30 05:52:54.975407 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.975400 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft" Oct 30 05:52:54.980627 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.980596 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j7pwr"] Oct 30 05:52:54.981371 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.980647 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2626d"] Oct 30 05:52:54.981371 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.980665 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-lq9h9"] Oct 30 05:52:54.981371 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.980681 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj"] Oct 30 05:52:54.981371 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.980697 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:54.981371 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.980701 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr"] Oct 30 05:52:54.981371 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.980986 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b9c9cf4b4-89zvd"] Oct 30 05:52:54.981371 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.981005 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft"] Oct 30 05:52:54.981371 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.981018 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w"] Oct 30 05:52:54.991151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.991128 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq"] Oct 30 05:52:54.991348 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.991183 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5gv9v"] Oct 30 05:52:54.991348 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.991214 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf"] Oct 30 05:52:54.993955 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.993936 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lrww4"] Oct 30 05:52:54.999355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.998677 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b994dfc85-2v9ww"] Oct 30 05:52:54.999355 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:54.998800 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:52:55.002710 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.002647 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5twq8"] Oct 30 05:52:55.011951 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.011928 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7ztmq"] Oct 30 05:52:55.012045 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.011977 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z"] Oct 30 05:52:55.028772 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.028745 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lrww4"] Oct 30 05:52:55.039748 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.039725 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:52:55.039855 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.039783 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 05:52:55.039855 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.039815 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:52:55.039958 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.039930 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 05:52:55.041210 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041188 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:52:55.041341 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041224 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 05:52:55.041341 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041273 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 05:52:55.041470 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041382 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:52:55.041470 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041415 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:52:55.041470 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041429 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:52:55.041470 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041450 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 05:52:55.041679 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041504 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 05:52:55.041679 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041523 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:52:55.041679 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041541 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:52:55.041679 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041556 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 05:52:55.041679 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041595 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 05:52:55.041936 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041740 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 05:52:55.041936 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041757 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 05:52:55.041936 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041812 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 05:52:55.041936 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041842 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 05:52:55.041936 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.041848 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 05:52:55.042246 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.042228 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:52:55.042360 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.042282 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:52:55.042360 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.042347 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 05:52:55.042458 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.042384 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:52:55.042511 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.042472 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:52:55.042643 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.042595 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:52:55.043451 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.043434 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:52:55.062657 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.062628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-snapshots\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.062766 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.062673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.062766 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.062747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.062848 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.062796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdslj\" (UniqueName: \"kubernetes.io/projected/67fb2af9-df3c-41a8-997c-e16de0871a3e-kube-api-access-wdslj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:52:55.062848 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.062833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-service-ca-bundle\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.062934 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.062876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.062934 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.062910 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-default-certificate\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.063032 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.062951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-serving-cert\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.063081 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-stats-auth\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.063133 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-trusted-ca\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.063133 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063106 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.063199 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn9vt\" (UniqueName: \"kubernetes.io/projected/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-kube-api-access-wn9vt\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:55.063235 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rx4n\" (UniqueName: \"kubernetes.io/projected/8cd01f41-97de-437c-aacf-29cb3668570b-kube-api-access-8rx4n\") pod \"kas-connection-checker-685895c7c9-j7pwr\" (UID: \"8cd01f41-97de-437c-aacf-29cb3668570b\") " pod="kube-system/kas-connection-checker-685895c7c9-j7pwr" Oct 30 05:52:55.063298 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-bound-sa-token\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.063398 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063385 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-bound-sa-token\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.063457 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.063518 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/391996f2-6666-4d62-9af2-1291a66828bf-config\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:52:55.063562 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6d3694c-d99c-43d1-be3f-6df16717e1de-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:52:55.063562 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8ltg\" (UniqueName: \"kubernetes.io/projected/ac9c2fad-cc58-4ec5-b169-271b6937af4b-kube-api-access-g8ltg\") pod \"volume-data-source-validator-58c9f5f46d-2ql6w\" (UID: \"ac9c2fad-cc58-4ec5-b169-271b6937af4b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w" Oct 30 05:52:55.063762 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-installation-pull-secrets\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.063804 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srpr\" (UniqueName: \"kubernetes.io/projected/391996f2-6666-4d62-9af2-1291a66828bf-kube-api-access-8srpr\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:52:55.063847 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:55.063954 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063937 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-certificates\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.064005 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.063981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/800e718b-7419-4df9-836a-cea3686ceda9-config-volume\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.064063 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064019 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzpmr\" (UniqueName: \"kubernetes.io/projected/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-kube-api-access-vzpmr\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.064120 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064077 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l52zw\" (UniqueName: \"kubernetes.io/projected/42155cb9-5b97-4aa0-8a18-70af60804e34-kube-api-access-l52zw\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:55.064120 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-registry-certificates\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.064220 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/184d6270-19ed-4e6b-80c6-9734584b82a1-ca-trust-extracted\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.064220 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhp65\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-kube-api-access-fhp65\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.064293 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6tct\" (UniqueName: \"kubernetes.io/projected/c7e5b861-eb47-4d8a-8974-90a2368489ed-kube-api-access-p6tct\") pod \"kas-connection-checker-685895c7c9-2626d\" (UID: \"c7e5b861-eb47-4d8a-8974-90a2368489ed\") " pod="kube-system/kas-connection-checker-685895c7c9-2626d" Oct 30 05:52:55.064369 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/391996f2-6666-4d62-9af2-1291a66828bf-serving-cert\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:52:55.064413 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f29d013-5eaf-42aa-860a-ba18193d7807-serving-cert\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.064480 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42155cb9-5b97-4aa0-8a18-70af60804e34-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:55.064539 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.064578 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-installation-pull-secrets\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.064578 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-image-registry-private-configuration\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.064647 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7k4q\" (UniqueName: \"kubernetes.io/projected/93ed3e75-7873-4300-be92-0cff9a0d63a7-kube-api-access-z7k4q\") pod \"ingress-canary-lrww4\" (UID: \"93ed3e75-7873-4300-be92-0cff9a0d63a7\") " pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:52:55.064647 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67fb2af9-df3c-41a8-997c-e16de0871a3e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:52:55.064728 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szdrr\" (UniqueName: \"kubernetes.io/projected/84346da8-55f2-4344-b7d0-dbfb2996f2e7-kube-api-access-szdrr\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.064728 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msw47\" (UniqueName: \"kubernetes.io/projected/800e718b-7419-4df9-836a-cea3686ceda9-kube-api-access-msw47\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.064792 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:55.064792 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:52:55.064866 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-image-registry-private-configuration\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.064866 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmlgr\" (UniqueName: \"kubernetes.io/projected/02b629c0-be01-461a-88b8-85a69dfad05b-kube-api-access-lmlgr\") pod \"kas-connection-checker-685895c7c9-74dvn\" (UID: \"02b629c0-be01-461a-88b8-85a69dfad05b\") " pod="kube-system/kas-connection-checker-685895c7c9-74dvn" Oct 30 05:52:55.064958 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-config\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.064958 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064905 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpbt\" (UniqueName: \"kubernetes.io/projected/8f29d013-5eaf-42aa-860a-ba18193d7807-kube-api-access-sfpbt\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.065020 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.064966 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbp7\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-kube-api-access-krbp7\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.065020 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.065004 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93ed3e75-7873-4300-be92-0cff9a0d63a7-cert\") pod \"ingress-canary-lrww4\" (UID: \"93ed3e75-7873-4300-be92-0cff9a0d63a7\") " pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:52:55.065090 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.065028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800e718b-7419-4df9-836a-cea3686ceda9-tmp-dir\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.065123 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.065114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00361e44-6e15-460e-88e7-1fd0398406e5-ca-trust-extracted\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.065165 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.065143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-trusted-ca\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.065223 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.065205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-trusted-ca\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.065283 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.065241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ngjf\" (UniqueName: \"kubernetes.io/projected/1b97e5c9-769c-4c97-8847-50a1d8ad613d-kube-api-access-2ngjf\") pod \"network-check-source-6f78d59f99-7jsft\" (UID: \"1b97e5c9-769c-4c97-8847-50a1d8ad613d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft" Oct 30 05:52:55.065283 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.065268 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67fb2af9-df3c-41a8-997c-e16de0871a3e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:52:55.065387 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.065326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:55.065387 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.065366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-tmp\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.066375 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066355 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 05:52:55.066528 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066514 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 05:52:55.066591 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066535 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 05:52:55.066684 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066664 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 05:52:55.066899 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066744 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6k8z9\"" Oct 30 05:52:55.066899 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066817 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:52:55.066899 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066832 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 05:52:55.066899 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066862 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 05:52:55.066899 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066748 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 05:52:55.066899 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066865 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r6cnz\"" Oct 30 05:52:55.067481 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066927 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 05:52:55.067481 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066966 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 05:52:55.067481 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066985 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 05:52:55.067481 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.066994 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:52:55.067481 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.067035 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 05:52:55.067481 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.067039 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 05:52:55.067481 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.067043 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-5fdqr\"" Oct 30 05:52:55.067481 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.067111 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntdkp\"" Oct 30 05:52:55.165864 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.165788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:52:55.165864 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.165818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-image-registry-private-configuration\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.165864 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.165837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlgr\" (UniqueName: \"kubernetes.io/projected/02b629c0-be01-461a-88b8-85a69dfad05b-kube-api-access-lmlgr\") pod \"kas-connection-checker-685895c7c9-74dvn\" (UID: \"02b629c0-be01-461a-88b8-85a69dfad05b\") " pod="kube-system/kas-connection-checker-685895c7c9-74dvn" Oct 30 05:52:55.165864 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.165853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-config\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.166163 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.165928 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:52:55.166163 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.165985 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert podName:d6d3694c-d99c-43d1-be3f-6df16717e1de nodeName:}" failed. No retries permitted until 2026-10-30 05:52:55.665969411 +0000 UTC m=+30.575702612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5twq8" (UID: "d6d3694c-d99c-43d1-be3f-6df16717e1de") : secret "networking-console-plugin-cert" not found Oct 30 05:52:55.166163 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.165931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpbt\" (UniqueName: \"kubernetes.io/projected/8f29d013-5eaf-42aa-860a-ba18193d7807-kube-api-access-sfpbt\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.166163 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbp7\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-kube-api-access-krbp7\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.166163 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93ed3e75-7873-4300-be92-0cff9a0d63a7-cert\") pod \"ingress-canary-lrww4\" (UID: \"93ed3e75-7873-4300-be92-0cff9a0d63a7\") " pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:52:55.166458 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800e718b-7419-4df9-836a-cea3686ceda9-tmp-dir\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.166458 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00361e44-6e15-460e-88e7-1fd0398406e5-ca-trust-extracted\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.166458 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-trusted-ca\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.166458 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-trusted-ca\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.166458 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166409 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ngjf\" (UniqueName: \"kubernetes.io/projected/1b97e5c9-769c-4c97-8847-50a1d8ad613d-kube-api-access-2ngjf\") pod \"network-check-source-6f78d59f99-7jsft\" (UID: \"1b97e5c9-769c-4c97-8847-50a1d8ad613d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft" Oct 30 05:52:55.166458 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67fb2af9-df3c-41a8-997c-e16de0871a3e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:52:55.166738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:55.166738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-tmp\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.166738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-snapshots\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.166738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.166738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.166738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166608 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdslj\" (UniqueName: \"kubernetes.io/projected/67fb2af9-df3c-41a8-997c-e16de0871a3e-kube-api-access-wdslj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:52:55.166738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-service-ca-bundle\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.166738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.166738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-default-certificate\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.166738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-serving-cert\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.166738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166730 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-config\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-stats-auth\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.166768 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-trusted-ca\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166816 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn9vt\" (UniqueName: \"kubernetes.io/projected/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-kube-api-access-wn9vt\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rx4n\" (UniqueName: \"kubernetes.io/projected/8cd01f41-97de-437c-aacf-29cb3668570b-kube-api-access-8rx4n\") pod \"kas-connection-checker-685895c7c9-j7pwr\" (UID: \"8cd01f41-97de-437c-aacf-29cb3668570b\") " pod="kube-system/kas-connection-checker-685895c7c9-j7pwr" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.166917 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls podName:42155cb9-5b97-4aa0-8a18-70af60804e34 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:55.666880904 +0000 UTC m=+30.576614110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkktj" (UID: "42155cb9-5b97-4aa0-8a18-70af60804e34") : secret "cluster-monitoring-operator-tls" not found Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.166952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-bound-sa-token\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167001 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00361e44-6e15-460e-88e7-1fd0398406e5-ca-trust-extracted\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-bound-sa-token\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/391996f2-6666-4d62-9af2-1291a66828bf-config\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6d3694c-d99c-43d1-be3f-6df16717e1de-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:52:55.167151 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8ltg\" (UniqueName: \"kubernetes.io/projected/ac9c2fad-cc58-4ec5-b169-271b6937af4b-kube-api-access-g8ltg\") pod \"volume-data-source-validator-58c9f5f46d-2ql6w\" (UID: \"ac9c2fad-cc58-4ec5-b169-271b6937af4b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-installation-pull-secrets\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srpr\" (UniqueName: \"kubernetes.io/projected/391996f2-6666-4d62-9af2-1291a66828bf-kube-api-access-8srpr\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-certificates\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/800e718b-7419-4df9-836a-cea3686ceda9-config-volume\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167347 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-trusted-ca\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpmr\" (UniqueName: \"kubernetes.io/projected/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-kube-api-access-vzpmr\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52zw\" (UniqueName: \"kubernetes.io/projected/42155cb9-5b97-4aa0-8a18-70af60804e34-kube-api-access-l52zw\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-registry-certificates\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/184d6270-19ed-4e6b-80c6-9734584b82a1-ca-trust-extracted\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhp65\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-kube-api-access-fhp65\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6tct\" (UniqueName: \"kubernetes.io/projected/c7e5b861-eb47-4d8a-8974-90a2368489ed-kube-api-access-p6tct\") pod \"kas-connection-checker-685895c7c9-2626d\" (UID: \"c7e5b861-eb47-4d8a-8974-90a2368489ed\") " pod="kube-system/kas-connection-checker-685895c7c9-2626d" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/391996f2-6666-4d62-9af2-1291a66828bf-serving-cert\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f29d013-5eaf-42aa-860a-ba18193d7807-serving-cert\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42155cb9-5b97-4aa0-8a18-70af60804e34-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:55.167722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-installation-pull-secrets\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-image-registry-private-configuration\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7k4q\" (UniqueName: \"kubernetes.io/projected/93ed3e75-7873-4300-be92-0cff9a0d63a7-kube-api-access-z7k4q\") pod \"ingress-canary-lrww4\" (UID: \"93ed3e75-7873-4300-be92-0cff9a0d63a7\") " pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67fb2af9-df3c-41a8-997c-e16de0871a3e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szdrr\" (UniqueName: \"kubernetes.io/projected/84346da8-55f2-4344-b7d0-dbfb2996f2e7-kube-api-access-szdrr\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167815 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-tmp\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msw47\" (UniqueName: \"kubernetes.io/projected/800e718b-7419-4df9-836a-cea3686ceda9-kube-api-access-msw47\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167922 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-trusted-ca\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.167970 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-trusted-ca\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.168130 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.168145 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9c9cf4b4-89zvd: secret "image-registry-tls" not found Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.168215 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls podName:00361e44-6e15-460e-88e7-1fd0398406e5 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:55.668199871 +0000 UTC m=+30.577933083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls") pod "image-registry-7b9c9cf4b4-89zvd" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5") : secret "image-registry-tls" not found Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.168249 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67fb2af9-df3c-41a8-997c-e16de0871a3e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.168271 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/184d6270-19ed-4e6b-80c6-9734584b82a1-ca-trust-extracted\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.168609 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.168321 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800e718b-7419-4df9-836a-cea3686ceda9-tmp-dir\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.169346 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.168563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-service-ca-bundle\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.169346 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.168677 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:55.668661156 +0000 UTC m=+30.578394376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : configmap references non-existent config key: service-ca.crt Oct 30 05:52:55.169346 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.168702 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-snapshots\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.169346 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.168750 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:52:55.169346 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.168786 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls podName:800e718b-7419-4df9-836a-cea3686ceda9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:55.668774003 +0000 UTC m=+30.578507219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls") pod "dns-default-5gv9v" (UID: "800e718b-7419-4df9-836a-cea3686ceda9") : secret "dns-default-metrics-tls" not found Oct 30 05:52:55.169346 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.169040 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:52:55.169346 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.169086 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls podName:6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef nodeName:}" failed. No retries permitted until 2026-10-30 05:52:55.669073029 +0000 UTC m=+30.578806245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-glfjr" (UID: "6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef") : secret "samples-operator-tls" not found Oct 30 05:52:55.169675 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.169277 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.171561 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.169604 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/391996f2-6666-4d62-9af2-1291a66828bf-config\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:52:55.171561 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.171375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-registry-certificates\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.171753 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.171647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-certificates\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.172120 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.172082 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-default-certificate\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.172120 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.172098 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 05:52:55.172120 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.172116 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-stats-auth\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.173452 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.172169 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:55.672146365 +0000 UTC m=+30.581879579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : secret "router-metrics-certs-default" not found Oct 30 05:52:55.173452 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.172237 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:55.173452 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.172767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-image-registry-private-configuration\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.173452 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.172834 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:52:55.173452 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.172850 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z: secret "image-registry-tls" not found Oct 30 05:52:55.173452 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.172909 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls podName:184d6270-19ed-4e6b-80c6-9734584b82a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:55.672891141 +0000 UTC m=+30.582624360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls") pod "image-registry-7f9d7c8b8f-gxn9z" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1") : secret "image-registry-tls" not found Oct 30 05:52:55.173452 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.173247 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93ed3e75-7873-4300-be92-0cff9a0d63a7-cert\") pod \"ingress-canary-lrww4\" (UID: \"93ed3e75-7873-4300-be92-0cff9a0d63a7\") " pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:52:55.173452 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.173290 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6d3694c-d99c-43d1-be3f-6df16717e1de-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:52:55.173452 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.173432 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67fb2af9-df3c-41a8-997c-e16de0871a3e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:52:55.173938 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.173533 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42155cb9-5b97-4aa0-8a18-70af60804e34-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:55.173938 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.173573 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-serving-cert\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.174239 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.174188 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/800e718b-7419-4df9-836a-cea3686ceda9-config-volume\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.178148 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.178101 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/391996f2-6666-4d62-9af2-1291a66828bf-serving-cert\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:52:55.178247 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.178193 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f29d013-5eaf-42aa-860a-ba18193d7807-serving-cert\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.178803 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.178772 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-installation-pull-secrets\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.178988 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.178972 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-installation-pull-secrets\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.179270 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.179250 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-image-registry-private-configuration\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.191083 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.191062 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpbt\" (UniqueName: \"kubernetes.io/projected/8f29d013-5eaf-42aa-860a-ba18193d7807-kube-api-access-sfpbt\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.199808 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.199784 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7k4q\" (UniqueName: \"kubernetes.io/projected/93ed3e75-7873-4300-be92-0cff9a0d63a7-kube-api-access-z7k4q\") pod \"ingress-canary-lrww4\" (UID: \"93ed3e75-7873-4300-be92-0cff9a0d63a7\") " pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:52:55.200664 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.200644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbp7\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-kube-api-access-krbp7\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.205675 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.205644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-bound-sa-token\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.205848 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.205829 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlgr\" (UniqueName: \"kubernetes.io/projected/02b629c0-be01-461a-88b8-85a69dfad05b-kube-api-access-lmlgr\") pod \"kas-connection-checker-685895c7c9-74dvn\" (UID: \"02b629c0-be01-461a-88b8-85a69dfad05b\") " pod="kube-system/kas-connection-checker-685895c7c9-74dvn" Oct 30 05:52:55.206099 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.206083 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ngjf\" (UniqueName: \"kubernetes.io/projected/1b97e5c9-769c-4c97-8847-50a1d8ad613d-kube-api-access-2ngjf\") pod \"network-check-source-6f78d59f99-7jsft\" (UID: \"1b97e5c9-769c-4c97-8847-50a1d8ad613d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft" Oct 30 05:52:55.208513 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.208492 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6tct\" (UniqueName: \"kubernetes.io/projected/c7e5b861-eb47-4d8a-8974-90a2368489ed-kube-api-access-p6tct\") pod \"kas-connection-checker-685895c7c9-2626d\" (UID: \"c7e5b861-eb47-4d8a-8974-90a2368489ed\") " pod="kube-system/kas-connection-checker-685895c7c9-2626d" Oct 30 05:52:55.209148 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.209128 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szdrr\" (UniqueName: \"kubernetes.io/projected/84346da8-55f2-4344-b7d0-dbfb2996f2e7-kube-api-access-szdrr\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.209245 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.209201 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8ltg\" (UniqueName: \"kubernetes.io/projected/ac9c2fad-cc58-4ec5-b169-271b6937af4b-kube-api-access-g8ltg\") pod \"volume-data-source-validator-58c9f5f46d-2ql6w\" (UID: \"ac9c2fad-cc58-4ec5-b169-271b6937af4b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w" Oct 30 05:52:55.210048 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.210029 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rx4n\" (UniqueName: \"kubernetes.io/projected/8cd01f41-97de-437c-aacf-29cb3668570b-kube-api-access-8rx4n\") pod \"kas-connection-checker-685895c7c9-j7pwr\" (UID: \"8cd01f41-97de-437c-aacf-29cb3668570b\") " pod="kube-system/kas-connection-checker-685895c7c9-j7pwr" Oct 30 05:52:55.210839 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.210817 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdslj\" (UniqueName: \"kubernetes.io/projected/67fb2af9-df3c-41a8-997c-e16de0871a3e-kube-api-access-wdslj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:52:55.214636 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.214618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msw47\" (UniqueName: \"kubernetes.io/projected/800e718b-7419-4df9-836a-cea3686ceda9-kube-api-access-msw47\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.214837 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.214816 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhp65\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-kube-api-access-fhp65\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.215027 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.215013 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn9vt\" (UniqueName: \"kubernetes.io/projected/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-kube-api-access-wn9vt\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:55.216763 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.216746 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpmr\" (UniqueName: \"kubernetes.io/projected/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-kube-api-access-vzpmr\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.219393 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.219375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-bound-sa-token\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.219761 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.219738 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srpr\" (UniqueName: \"kubernetes.io/projected/391996f2-6666-4d62-9af2-1291a66828bf-kube-api-access-8srpr\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:52:55.220261 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.220245 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52zw\" (UniqueName: \"kubernetes.io/projected/42155cb9-5b97-4aa0-8a18-70af60804e34-kube-api-access-l52zw\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:55.232709 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.232690 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-74dvn" Oct 30 05:52:55.242063 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.242043 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j7pwr" Oct 30 05:52:55.248885 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.248871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2626d" Oct 30 05:52:55.259775 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.259753 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:52:55.277420 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.277398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:52:55.284498 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.284475 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:52:55.334205 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.332613 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:52:55.335579 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.335544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w" Oct 30 05:52:55.369108 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.367793 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft" Oct 30 05:52:55.375753 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.375548 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:52:55.619266 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.619234 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2626d"] Oct 30 05:52:55.622752 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.622731 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j7pwr"] Oct 30 05:52:55.625247 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.625225 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-74dvn"] Oct 30 05:52:55.627792 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.627731 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf"] Oct 30 05:52:55.629022 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:55.628992 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e5b861_eb47_4d8a_8974_90a2368489ed.slice/crio-7951402aeba57244ffbe2add8b3ef385c13e12c15f4417740d8c6a7b3e06f75d WatchSource:0}: Error finding container 7951402aeba57244ffbe2add8b3ef385c13e12c15f4417740d8c6a7b3e06f75d: Status 404 returned error can't find the container with id 7951402aeba57244ffbe2add8b3ef385c13e12c15f4417740d8c6a7b3e06f75d Oct 30 05:52:55.629748 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:55.629727 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd01f41_97de_437c_aacf_29cb3668570b.slice/crio-50fe92da315b6dc10e5250bca9dcb85546c6e5aa696bf1c3dff174ef9e1bc17e WatchSource:0}: Error finding container 50fe92da315b6dc10e5250bca9dcb85546c6e5aa696bf1c3dff174ef9e1bc17e: Status 404 returned error can't find the container with id 50fe92da315b6dc10e5250bca9dcb85546c6e5aa696bf1c3dff174ef9e1bc17e Oct 30 05:52:55.631385 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:55.631085 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b629c0_be01_461a_88b8_85a69dfad05b.slice/crio-4c453175eff516c698e66a1b7fca3d1cc5b61ff55636b4bf3f2082ccaa4ff1a9 WatchSource:0}: Error finding container 4c453175eff516c698e66a1b7fca3d1cc5b61ff55636b4bf3f2082ccaa4ff1a9: Status 404 returned error can't find the container with id 4c453175eff516c698e66a1b7fca3d1cc5b61ff55636b4bf3f2082ccaa4ff1a9 Oct 30 05:52:55.631747 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:55.631729 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fb2af9_df3c_41a8_997c_e16de0871a3e.slice/crio-303f050fdbf7b3c43f865d3732ffc24453c1c8fcdc7146738e23217094f5a4f4 WatchSource:0}: Error finding container 303f050fdbf7b3c43f865d3732ffc24453c1c8fcdc7146738e23217094f5a4f4: Status 404 returned error can't find the container with id 303f050fdbf7b3c43f865d3732ffc24453c1c8fcdc7146738e23217094f5a4f4 Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.674805 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.674852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.674883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.674917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.674959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.675013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.675096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.675140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675281 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675366 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert podName:d6d3694c-d99c-43d1-be3f-6df16717e1de nodeName:}" failed. No retries permitted until 2026-10-30 05:52:56.675348176 +0000 UTC m=+31.585081378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5twq8" (UID: "d6d3694c-d99c-43d1-be3f-6df16717e1de") : secret "networking-console-plugin-cert" not found Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675473 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675489 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:56.675458814 +0000 UTC m=+31.585192028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : configmap references non-existent config key: service-ca.crt Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675517 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls podName:6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef nodeName:}" failed. No retries permitted until 2026-10-30 05:52:56.675505562 +0000 UTC m=+31.585238762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-glfjr" (UID: "6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef") : secret "samples-operator-tls" not found Oct 30 05:52:55.675616 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675610 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:52:55.676397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675660 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls podName:800e718b-7419-4df9-836a-cea3686ceda9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:56.675645126 +0000 UTC m=+31.585378328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls") pod "dns-default-5gv9v" (UID: "800e718b-7419-4df9-836a-cea3686ceda9") : secret "dns-default-metrics-tls" not found Oct 30 05:52:55.676397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675731 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 05:52:55.676397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675773 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls podName:42155cb9-5b97-4aa0-8a18-70af60804e34 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:56.675756411 +0000 UTC m=+31.585489620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkktj" (UID: "42155cb9-5b97-4aa0-8a18-70af60804e34") : secret "cluster-monitoring-operator-tls" not found Oct 30 05:52:55.676397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675905 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:52:55.676397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675919 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9c9cf4b4-89zvd: secret "image-registry-tls" not found Oct 30 05:52:55.676397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.675969 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls podName:00361e44-6e15-460e-88e7-1fd0398406e5 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:56.675956151 +0000 UTC m=+31.585689368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls") pod "image-registry-7b9c9cf4b4-89zvd" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5") : secret "image-registry-tls" not found Oct 30 05:52:55.676397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.676024 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:52:55.676397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.676035 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z: secret "image-registry-tls" not found Oct 30 05:52:55.676397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.676091 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls podName:184d6270-19ed-4e6b-80c6-9734584b82a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:56.676072537 +0000 UTC m=+31.585805752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls") pod "image-registry-7f9d7c8b8f-gxn9z" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1") : secret "image-registry-tls" not found Oct 30 05:52:55.676397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.676142 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 05:52:55.676397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:55.676183 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:56.676165675 +0000 UTC m=+31.585898875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : secret "router-metrics-certs-default" not found Oct 30 05:52:55.681548 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.681452 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7ztmq"] Oct 30 05:52:55.686589 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:55.686560 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9c2d4d_67e5_4c4d_8294_86ae9050232a.slice/crio-d8db7439dddd8b2f53cd4e6ac413734c8079dfd5e6267ba1170fd427879760fc WatchSource:0}: Error finding container d8db7439dddd8b2f53cd4e6ac413734c8079dfd5e6267ba1170fd427879760fc: Status 404 returned error can't find the container with id d8db7439dddd8b2f53cd4e6ac413734c8079dfd5e6267ba1170fd427879760fc Oct 30 05:52:55.691677 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.691618 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-lq9h9"] Oct 30 05:52:55.696078 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:55.696049 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f29d013_5eaf_42aa_860a_ba18193d7807.slice/crio-937c4a9525ad300b9baf2e749750324f985f733829dcf4fe937199dd621e013c WatchSource:0}: Error finding container 937c4a9525ad300b9baf2e749750324f985f733829dcf4fe937199dd621e013c: Status 404 returned error can't find the container with id 937c4a9525ad300b9baf2e749750324f985f733829dcf4fe937199dd621e013c Oct 30 05:52:55.741544 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.741515 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft"] Oct 30 05:52:55.745674 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.745652 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq"] Oct 30 05:52:55.746395 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:55.746373 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b97e5c9_769c_4c97_8847_50a1d8ad613d.slice/crio-9ab94d7e66d56937833d15fcb51b0c16944de28897faecdfd3d843d6aa885ca8 WatchSource:0}: Error finding container 9ab94d7e66d56937833d15fcb51b0c16944de28897faecdfd3d843d6aa885ca8: Status 404 returned error can't find the container with id 9ab94d7e66d56937833d15fcb51b0c16944de28897faecdfd3d843d6aa885ca8 Oct 30 05:52:55.752018 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:55.751797 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391996f2_6666_4d62_9af2_1291a66828bf.slice/crio-6167377e81c1d4b4f582f023dc700b09a8340d50657f21a7b7a131a41dccd9f3 WatchSource:0}: Error finding container 6167377e81c1d4b4f582f023dc700b09a8340d50657f21a7b7a131a41dccd9f3: Status 404 returned error can't find the container with id 6167377e81c1d4b4f582f023dc700b09a8340d50657f21a7b7a131a41dccd9f3 Oct 30 05:52:55.783163 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.783138 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w"] Oct 30 05:52:55.785664 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:55.785641 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9c2fad_cc58_4ec5_b169_271b6937af4b.slice/crio-3979a4dbf45de6174e65b75182d05294cd60f1fab756ae5cec10d8f469f33e21 WatchSource:0}: Error finding container 3979a4dbf45de6174e65b75182d05294cd60f1fab756ae5cec10d8f469f33e21: Status 404 returned error can't find the container with id 3979a4dbf45de6174e65b75182d05294cd60f1fab756ae5cec10d8f469f33e21 Oct 30 05:52:55.811289 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.811266 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lrww4"] Oct 30 05:52:55.814450 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:55.814422 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ed3e75_7873_4300_be92_0cff9a0d63a7.slice/crio-1dec71dde8963b69ee2b5f03cd1adb9d18ccc4d80f64dd47757147179b57f692 WatchSource:0}: Error finding container 1dec71dde8963b69ee2b5f03cd1adb9d18ccc4d80f64dd47757147179b57f692: Status 404 returned error can't find the container with id 1dec71dde8963b69ee2b5f03cd1adb9d18ccc4d80f64dd47757147179b57f692 Oct 30 05:52:55.861286 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.861256 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w" event={"ID":"ac9c2fad-cc58-4ec5-b169-271b6937af4b","Type":"ContainerStarted","Data":"3979a4dbf45de6174e65b75182d05294cd60f1fab756ae5cec10d8f469f33e21"} Oct 30 05:52:55.862720 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.862694 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j7pwr" event={"ID":"8cd01f41-97de-437c-aacf-29cb3668570b","Type":"ContainerStarted","Data":"68a12173cfaf8cee309aef98e5b25408c01e96ab81e0f4ada57c70a47d66b829"} Oct 30 05:52:55.862819 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.862729 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j7pwr" event={"ID":"8cd01f41-97de-437c-aacf-29cb3668570b","Type":"ContainerStarted","Data":"50fe92da315b6dc10e5250bca9dcb85546c6e5aa696bf1c3dff174ef9e1bc17e"} Oct 30 05:52:55.863910 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.863890 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lrww4" event={"ID":"93ed3e75-7873-4300-be92-0cff9a0d63a7","Type":"ContainerStarted","Data":"1dec71dde8963b69ee2b5f03cd1adb9d18ccc4d80f64dd47757147179b57f692"} Oct 30 05:52:55.864987 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.864969 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lq9h9" event={"ID":"8f29d013-5eaf-42aa-860a-ba18193d7807","Type":"ContainerStarted","Data":"937c4a9525ad300b9baf2e749750324f985f733829dcf4fe937199dd621e013c"} Oct 30 05:52:55.866393 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.866374 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-74dvn" event={"ID":"02b629c0-be01-461a-88b8-85a69dfad05b","Type":"ContainerStarted","Data":"018e21a1c5111266bdb3eb19b2305eddf0f6018aaecc4ef272dc7c65389be948"} Oct 30 05:52:55.866489 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.866402 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-74dvn" event={"ID":"02b629c0-be01-461a-88b8-85a69dfad05b","Type":"ContainerStarted","Data":"4c453175eff516c698e66a1b7fca3d1cc5b61ff55636b4bf3f2082ccaa4ff1a9"} Oct 30 05:52:55.867551 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.867531 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft" event={"ID":"1b97e5c9-769c-4c97-8847-50a1d8ad613d","Type":"ContainerStarted","Data":"9ab94d7e66d56937833d15fcb51b0c16944de28897faecdfd3d843d6aa885ca8"} Oct 30 05:52:55.869451 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.869416 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" event={"ID":"67fb2af9-df3c-41a8-997c-e16de0871a3e","Type":"ContainerStarted","Data":"303f050fdbf7b3c43f865d3732ffc24453c1c8fcdc7146738e23217094f5a4f4"} Oct 30 05:52:55.870708 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.870678 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" event={"ID":"391996f2-6666-4d62-9af2-1291a66828bf","Type":"ContainerStarted","Data":"6167377e81c1d4b4f582f023dc700b09a8340d50657f21a7b7a131a41dccd9f3"} Oct 30 05:52:55.871800 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.871779 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" event={"ID":"cb9c2d4d-67e5-4c4d-8294-86ae9050232a","Type":"ContainerStarted","Data":"d8db7439dddd8b2f53cd4e6ac413734c8079dfd5e6267ba1170fd427879760fc"} Oct 30 05:52:55.872957 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.872938 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2626d" event={"ID":"c7e5b861-eb47-4d8a-8974-90a2368489ed","Type":"ContainerStarted","Data":"a6dcf33ef17ee72abc75971269a29674d01b9f0fe2ccb833e07405f621082575"} Oct 30 05:52:55.873040 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.872972 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2626d" event={"ID":"c7e5b861-eb47-4d8a-8974-90a2368489ed","Type":"ContainerStarted","Data":"7951402aeba57244ffbe2add8b3ef385c13e12c15f4417740d8c6a7b3e06f75d"} Oct 30 05:52:55.902509 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.902473 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-j7pwr" podStartSLOduration=331.902462739 podStartE2EDuration="5m31.902462739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:55.901293633 +0000 UTC m=+30.811026854" watchObservedRunningTime="2026-10-30 05:52:55.902462739 +0000 UTC m=+30.812195960" Oct 30 05:52:55.943288 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.943219 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-74dvn" podStartSLOduration=331.943208021 podStartE2EDuration="5m31.943208021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:55.942806571 +0000 UTC m=+30.852539792" watchObservedRunningTime="2026-10-30 05:52:55.943208021 +0000 UTC m=+30.852941241" Oct 30 05:52:55.987561 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:55.987515 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-2626d" podStartSLOduration=331.987500669 podStartE2EDuration="5m31.987500669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:55.986837993 +0000 UTC m=+30.896571216" watchObservedRunningTime="2026-10-30 05:52:55.987500669 +0000 UTC m=+30.897233893" Oct 30 05:52:56.589432 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.589401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:56.589950 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.589928 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:56.590409 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.590392 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:56.605196 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.605177 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l4lh5\"" Oct 30 05:52:56.605328 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.605256 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-99gsm\"" Oct 30 05:52:56.605373 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.605339 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xln8v\"" Oct 30 05:52:56.605373 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.605364 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:52:56.605469 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.605423 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.685367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.685488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.685532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.685592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.685628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.685657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.685687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:56.685714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.685824 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.685879 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:58.685867016 +0000 UTC m=+33.595600228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : secret "router-metrics-certs-default" not found Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686187 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686243 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls podName:6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef nodeName:}" failed. No retries permitted until 2026-10-30 05:52:58.686228525 +0000 UTC m=+33.595961738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-glfjr" (UID: "6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef") : secret "samples-operator-tls" not found Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686324 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686336 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z: secret "image-registry-tls" not found Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686360 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls podName:184d6270-19ed-4e6b-80c6-9734584b82a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:58.686352693 +0000 UTC m=+33.596085893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls") pod "image-registry-7f9d7c8b8f-gxn9z" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1") : secret "image-registry-tls" not found Oct 30 05:52:56.686807 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686420 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:52:56.687734 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686449 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert podName:d6d3694c-d99c-43d1-be3f-6df16717e1de nodeName:}" failed. No retries permitted until 2026-10-30 05:52:58.686442624 +0000 UTC m=+33.596175836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5twq8" (UID: "d6d3694c-d99c-43d1-be3f-6df16717e1de") : secret "networking-console-plugin-cert" not found Oct 30 05:52:56.687734 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686495 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 05:52:56.687734 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686526 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls podName:42155cb9-5b97-4aa0-8a18-70af60804e34 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:58.686515486 +0000 UTC m=+33.596248699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkktj" (UID: "42155cb9-5b97-4aa0-8a18-70af60804e34") : secret "cluster-monitoring-operator-tls" not found Oct 30 05:52:56.687734 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686591 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:58.686581081 +0000 UTC m=+33.596314282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : configmap references non-existent config key: service-ca.crt Oct 30 05:52:56.687734 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686653 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:52:56.687734 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686681 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls podName:800e718b-7419-4df9-836a-cea3686ceda9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:58.686671998 +0000 UTC m=+33.596405203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls") pod "dns-default-5gv9v" (UID: "800e718b-7419-4df9-836a-cea3686ceda9") : secret "dns-default-metrics-tls" not found Oct 30 05:52:56.687734 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686739 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:52:56.687734 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686750 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9c9cf4b4-89zvd: secret "image-registry-tls" not found Oct 30 05:52:56.687734 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:56.686776 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls podName:00361e44-6e15-460e-88e7-1fd0398406e5 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:58.686770272 +0000 UTC m=+33.596503484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls") pod "image-registry-7b9c9cf4b4-89zvd" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5") : secret "image-registry-tls" not found Oct 30 05:52:58.712560 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:58.712523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:58.712581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:58.712642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:58.712670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.712692 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.712710 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z: secret "image-registry-tls" not found Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.712753 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.712768 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls podName:184d6270-19ed-4e6b-80c6-9734584b82a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:02.712750367 +0000 UTC m=+37.622483588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls") pod "image-registry-7f9d7c8b8f-gxn9z" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1") : secret "image-registry-tls" not found Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.712784 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls podName:800e718b-7419-4df9-836a-cea3686ceda9 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:02.712774822 +0000 UTC m=+37.622508022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls") pod "dns-default-5gv9v" (UID: "800e718b-7419-4df9-836a-cea3686ceda9") : secret "dns-default-metrics-tls" not found Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.712826 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.712854 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert podName:d6d3694c-d99c-43d1-be3f-6df16717e1de nodeName:}" failed. No retries permitted until 2026-10-30 05:53:02.712845199 +0000 UTC m=+37.622578400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5twq8" (UID: "d6d3694c-d99c-43d1-be3f-6df16717e1de") : secret "networking-console-plugin-cert" not found Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.712902 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.712929 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls podName:42155cb9-5b97-4aa0-8a18-70af60804e34 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:02.712920421 +0000 UTC m=+37.622653620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkktj" (UID: "42155cb9-5b97-4aa0-8a18-70af60804e34") : secret "cluster-monitoring-operator-tls" not found Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.713004 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:02.712995155 +0000 UTC m=+37.622728369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : configmap references non-existent config key: service-ca.crt Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:58.712692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:52:58.713159 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:58.713043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:52:58.713839 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:58.713083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:52:58.713839 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:58.713134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:52:58.713839 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.713177 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:52:58.713839 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.713189 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9c9cf4b4-89zvd: secret "image-registry-tls" not found Oct 30 05:52:58.713839 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.713225 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls podName:00361e44-6e15-460e-88e7-1fd0398406e5 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:02.713213885 +0000 UTC m=+37.622947098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls") pod "image-registry-7b9c9cf4b4-89zvd" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5") : secret "image-registry-tls" not found Oct 30 05:52:58.713839 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.713235 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:52:58.713839 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.713267 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls podName:6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef nodeName:}" failed. No retries permitted until 2026-10-30 05:53:02.713256341 +0000 UTC m=+37.622989547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-glfjr" (UID: "6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef") : secret "samples-operator-tls" not found Oct 30 05:52:58.713839 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.713277 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 05:52:58.713839 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:58.713333 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:02.713323207 +0000 UTC m=+37.623056420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : secret "router-metrics-certs-default" not found Oct 30 05:52:58.893413 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:58.892603 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w" event={"ID":"ac9c2fad-cc58-4ec5-b169-271b6937af4b","Type":"ContainerStarted","Data":"52c36dfd282d37f8a1c0d4377e36893380cdfa5ce9e454de1de7209d564d2561"} Oct 30 05:52:58.933937 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:58.933880 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w" podStartSLOduration=298.835019622 podStartE2EDuration="5m0.933866491s" totalImagesPullingTime="2.098846869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:58.933541163 +0000 UTC m=+33.843274384" watchObservedRunningTime="2026-10-30 05:52:58.933866491 +0000 UTC m=+33.843599711" Oct 30 05:52:59.320762 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.320680 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:59.320762 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.320758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:52:59.320991 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:59.320897 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:52:59.320991 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:52:59.320943 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:53:31.320931124 +0000 UTC m=+66.230664336 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : secret "metrics-daemon-secret" not found Oct 30 05:52:59.327037 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.326975 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:59.362263 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.362229 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:52:59.422109 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.422078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xjq\" (UniqueName: \"kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq\") pod \"network-check-target-xsrdq\" (UID: \"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8\") " pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:59.425519 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.425493 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xjq\" (UniqueName: \"kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq\") pod \"network-check-target-xsrdq\" (UID: \"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8\") " pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:59.688511 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.688212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:52:59.824077 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.823893 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bnv8q"] Oct 30 05:52:59.864014 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:52:59.863979 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44df296_4186_4c40_81de_1ed06c15955e.slice/crio-a7e8d3e8247283e32225098487ec34520a2a90cb84181d0b971e199a64baab0e WatchSource:0}: Error finding container a7e8d3e8247283e32225098487ec34520a2a90cb84181d0b971e199a64baab0e: Status 404 returned error can't find the container with id a7e8d3e8247283e32225098487ec34520a2a90cb84181d0b971e199a64baab0e Oct 30 05:52:59.896633 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.896597 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bnv8q" event={"ID":"d44df296-4186-4c40-81de-1ed06c15955e","Type":"ContainerStarted","Data":"a7e8d3e8247283e32225098487ec34520a2a90cb84181d0b971e199a64baab0e"} Oct 30 05:52:59.899014 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.898993 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/0.log" Oct 30 05:52:59.899146 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.899029 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb9c2d4d-67e5-4c4d-8294-86ae9050232a" containerID="d919aaec6aa3995ba3abe2b30bbdbdca3e778d4c9e15f6740a37e53337ed0e73" exitCode=255 Oct 30 05:52:59.899763 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.899745 2578 scope.go:122] "RemoveContainer" containerID="d919aaec6aa3995ba3abe2b30bbdbdca3e778d4c9e15f6740a37e53337ed0e73" Oct 30 05:52:59.900009 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:52:59.899985 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" event={"ID":"cb9c2d4d-67e5-4c4d-8294-86ae9050232a","Type":"ContainerDied","Data":"d919aaec6aa3995ba3abe2b30bbdbdca3e778d4c9e15f6740a37e53337ed0e73"} Oct 30 05:53:00.032084 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:00.032046 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsrdq"] Oct 30 05:53:00.172827 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:53:00.172735 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbf9447_7ab2_49e9_ae50_99ec6a57e3c8.slice/crio-5fa96e71655df633bb5f03b03d39f54c418e1841d26a7b50e96f240648c751f5 WatchSource:0}: Error finding container 5fa96e71655df633bb5f03b03d39f54c418e1841d26a7b50e96f240648c751f5: Status 404 returned error can't find the container with id 5fa96e71655df633bb5f03b03d39f54c418e1841d26a7b50e96f240648c751f5 Oct 30 05:53:00.913649 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:00.913607 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" event={"ID":"67fb2af9-df3c-41a8-997c-e16de0871a3e","Type":"ContainerStarted","Data":"62e3999d9e3d7a38271da565e83facd860d9fd6ed78b657265d35e365556b854"} Oct 30 05:53:00.915934 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:00.915898 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 05:53:00.916456 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:00.916439 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/0.log" Oct 30 05:53:00.916574 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:00.916474 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb9c2d4d-67e5-4c4d-8294-86ae9050232a" containerID="a1b0e58f3452ccffe96b15479c7192f31726ad35f4c7c369ea791c73d61053fe" exitCode=255 Oct 30 05:53:00.916643 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:00.916573 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" event={"ID":"cb9c2d4d-67e5-4c4d-8294-86ae9050232a","Type":"ContainerDied","Data":"a1b0e58f3452ccffe96b15479c7192f31726ad35f4c7c369ea791c73d61053fe"} Oct 30 05:53:00.916643 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:00.916633 2578 scope.go:122] "RemoveContainer" containerID="d919aaec6aa3995ba3abe2b30bbdbdca3e778d4c9e15f6740a37e53337ed0e73" Oct 30 05:53:00.916898 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:00.916881 2578 scope.go:122] "RemoveContainer" containerID="a1b0e58f3452ccffe96b15479c7192f31726ad35f4c7c369ea791c73d61053fe" Oct 30 05:53:00.917106 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:00.917085 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7ztmq_openshift-console-operator(cb9c2d4d-67e5-4c4d-8294-86ae9050232a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" podUID="cb9c2d4d-67e5-4c4d-8294-86ae9050232a" Oct 30 05:53:00.920043 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:00.919925 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lq9h9" event={"ID":"8f29d013-5eaf-42aa-860a-ba18193d7807","Type":"ContainerStarted","Data":"428158fcd4240a908e4f2817841dfc060be33bea1c07c2c6a40c3a7666541d8d"} Oct 30 05:53:00.922019 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:00.921992 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsrdq" event={"ID":"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8","Type":"ContainerStarted","Data":"5fa96e71655df633bb5f03b03d39f54c418e1841d26a7b50e96f240648c751f5"} Oct 30 05:53:00.958911 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:00.958868 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" podStartSLOduration=302.174935246 podStartE2EDuration="5m6.958851259s" totalImagesPullingTime="4.783916013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:00.958205841 +0000 UTC m=+35.867939063" watchObservedRunningTime="2026-10-30 05:53:00.958851259 +0000 UTC m=+35.868584483" Oct 30 05:53:01.047326 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.047240 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-lq9h9" podStartSLOduration=302.318444755 podStartE2EDuration="5m7.047221993s" totalImagesPullingTime="4.728777238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:01.046407209 +0000 UTC m=+35.956140430" watchObservedRunningTime="2026-10-30 05:53:01.047221993 +0000 UTC m=+35.956955217" Oct 30 05:53:01.442771 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.442737 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dtvjb"] Oct 30 05:53:01.477966 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.476855 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dtvjb"] Oct 30 05:53:01.477966 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.476990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.496262 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.496196 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x8h66\"" Oct 30 05:53:01.499864 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.499294 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:53:01.547148 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.545818 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-ghpdk-jn8g9"] Oct 30 05:53:01.547148 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.546642 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfq7\" (UniqueName: \"kubernetes.io/projected/a792a513-44ef-43d1-a48c-e17d8f939042-kube-api-access-lmfq7\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.547148 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.546773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.547148 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.546845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a792a513-44ef-43d1-a48c-e17d8f939042-crio-socket\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.547148 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.546945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a792a513-44ef-43d1-a48c-e17d8f939042-data-volume\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.547148 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.546978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.555917 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.555884 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:53:01.568035 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.567656 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.579019 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.578954 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ghpdk-jn8g9"] Oct 30 05:53:01.587218 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.587195 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dk4qd\"" Oct 30 05:53:01.640971 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.639613 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts"] Oct 30 05:53:01.647722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.647549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-trusted-ca-bundle\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.647722 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.647697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-service-ca-bundle\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.647914 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.647842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a792a513-44ef-43d1-a48c-e17d8f939042-data-volume\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.647914 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.647894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.647986 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.647940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfq7\" (UniqueName: \"kubernetes.io/projected/a792a513-44ef-43d1-a48c-e17d8f939042-kube-api-access-lmfq7\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.647986 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.647981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-archives-path\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.648104 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.648088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.648235 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.648120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgl2n\" (UniqueName: \"kubernetes.io/projected/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-kube-api-access-hgl2n\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.648235 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.648149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a792a513-44ef-43d1-a48c-e17d8f939042-crio-socket\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.648454 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.648430 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a792a513-44ef-43d1-a48c-e17d8f939042-crio-socket\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.648748 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.648733 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a792a513-44ef-43d1-a48c-e17d8f939042-data-volume\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.648811 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:01.648798 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:53:01.648902 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:01.648855 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:02.148838271 +0000 UTC m=+37.058571475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : configmap "kube-rbac-proxy" not found Oct 30 05:53:01.649384 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:01.649365 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:01.649520 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:01.649417 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:02.149403403 +0000 UTC m=+37.059136616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:01.660705 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.659877 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr"] Oct 30 05:53:01.660705 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.659998 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts" Oct 30 05:53:01.677944 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.677918 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wkwzp\"" Oct 30 05:53:01.683989 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.683964 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts"] Oct 30 05:53:01.684124 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.684091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr" Oct 30 05:53:01.695566 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.692583 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfq7\" (UniqueName: \"kubernetes.io/projected/a792a513-44ef-43d1-a48c-e17d8f939042-kube-api-access-lmfq7\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:01.712211 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.712180 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr"] Oct 30 05:53:01.723941 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.723914 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:01.724089 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.724005 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 05:53:01.748899 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.748838 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgl2n\" (UniqueName: \"kubernetes.io/projected/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-kube-api-access-hgl2n\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.748899 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.748898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhgdv\" (UniqueName: \"kubernetes.io/projected/6c72f864-73b2-48b6-ba6c-fe16323a2d18-kube-api-access-nhgdv\") pod \"migrator-66df5d496b-mjhts\" (UID: \"6c72f864-73b2-48b6-ba6c-fe16323a2d18\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts" Oct 30 05:53:01.749135 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.748959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-trusted-ca-bundle\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.749135 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.748996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-service-ca-bundle\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.749135 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.749038 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftrml\" (UniqueName: \"kubernetes.io/projected/f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9-kube-api-access-ftrml\") pod \"migrator-66df5d496b-7qmpr\" (UID: \"f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr" Oct 30 05:53:01.749135 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.749077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-archives-path\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.749575 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.749555 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-archives-path\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.749788 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.749758 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-service-ca-bundle\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.750387 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.750363 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-trusted-ca-bundle\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.783824 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.783789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgl2n\" (UniqueName: \"kubernetes.io/projected/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-kube-api-access-hgl2n\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.850147 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.850107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhgdv\" (UniqueName: \"kubernetes.io/projected/6c72f864-73b2-48b6-ba6c-fe16323a2d18-kube-api-access-nhgdv\") pod \"migrator-66df5d496b-mjhts\" (UID: \"6c72f864-73b2-48b6-ba6c-fe16323a2d18\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts" Oct 30 05:53:01.850559 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.850205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrml\" (UniqueName: \"kubernetes.io/projected/f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9-kube-api-access-ftrml\") pod \"migrator-66df5d496b-7qmpr\" (UID: \"f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr" Oct 30 05:53:01.881732 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.881671 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrml\" (UniqueName: \"kubernetes.io/projected/f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9-kube-api-access-ftrml\") pod \"migrator-66df5d496b-7qmpr\" (UID: \"f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr" Oct 30 05:53:01.882352 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.882327 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhgdv\" (UniqueName: \"kubernetes.io/projected/6c72f864-73b2-48b6-ba6c-fe16323a2d18-kube-api-access-nhgdv\") pod \"migrator-66df5d496b-mjhts\" (UID: \"6c72f864-73b2-48b6-ba6c-fe16323a2d18\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts" Oct 30 05:53:01.895622 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.895595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:01.930604 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.930542 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lrww4" event={"ID":"93ed3e75-7873-4300-be92-0cff9a0d63a7","Type":"ContainerStarted","Data":"6c2648fbd8c95f1d37d7745fb45a4f280d12202b7ce0e2db240d3661788ebd90"} Oct 30 05:53:01.934759 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.934731 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft" event={"ID":"1b97e5c9-769c-4c97-8847-50a1d8ad613d","Type":"ContainerStarted","Data":"42fe67c6408046b6976ff919b3e6f95fb19f04c3ef35491302e236b12160e9e9"} Oct 30 05:53:01.936966 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.936718 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" event={"ID":"391996f2-6666-4d62-9af2-1291a66828bf","Type":"ContainerStarted","Data":"156b79b7901aa86644221a56d1ae524c2bd153ca8c64deb1d8ea0a6fd96ea9ea"} Oct 30 05:53:01.947206 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.942626 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 05:53:01.947206 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.943590 2578 scope.go:122] "RemoveContainer" containerID="a1b0e58f3452ccffe96b15479c7192f31726ad35f4c7c369ea791c73d61053fe" Oct 30 05:53:01.947206 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:01.943842 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7ztmq_openshift-console-operator(cb9c2d4d-67e5-4c4d-8294-86ae9050232a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" podUID="cb9c2d4d-67e5-4c4d-8294-86ae9050232a" Oct 30 05:53:01.975010 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.974832 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lrww4" podStartSLOduration=2.724558736 podStartE2EDuration="7.97481048s" totalImagesPullingTime="5.250251744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:01.97265807 +0000 UTC m=+36.882391281" watchObservedRunningTime="2026-10-30 05:53:01.97481048 +0000 UTC m=+36.884543704" Oct 30 05:53:01.991111 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:01.990864 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts" Oct 30 05:53:02.024292 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.023873 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft" podStartSLOduration=294.355519022 podStartE2EDuration="5m0.023851641s" totalImagesPullingTime="5.668332619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:02.02002664 +0000 UTC m=+36.929759864" watchObservedRunningTime="2026-10-30 05:53:02.023851641 +0000 UTC m=+36.933584862" Oct 30 05:53:02.024292 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.024149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr" Oct 30 05:53:02.118157 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.118045 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" podStartSLOduration=299.817234477 podStartE2EDuration="5m5.118024601s" totalImagesPullingTime="5.300790124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:02.117592299 +0000 UTC m=+37.027325522" watchObservedRunningTime="2026-10-30 05:53:02.118024601 +0000 UTC m=+37.027757824" Oct 30 05:53:02.152939 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.152899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:02.153092 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.153049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:02.153204 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.153182 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:53:02.153338 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.153293 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:03.153273264 +0000 UTC m=+38.063006469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : configmap "kube-rbac-proxy" not found Oct 30 05:53:02.153428 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.153409 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:02.153479 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.153463 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:03.153450371 +0000 UTC m=+38.063183576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:02.257112 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.257043 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ghpdk-jn8g9"] Oct 30 05:53:02.259290 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:53:02.259252 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc92757_4a0b_44d6_9183_9d28c1b0d5e9.slice/crio-7e40cc373f4722e46eeee669f2b7f968458827ab0e49b430efa82d3190db4463 WatchSource:0}: Error finding container 7e40cc373f4722e46eeee669f2b7f968458827ab0e49b430efa82d3190db4463: Status 404 returned error can't find the container with id 7e40cc373f4722e46eeee669f2b7f968458827ab0e49b430efa82d3190db4463 Oct 30 05:53:02.341278 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.341249 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts"] Oct 30 05:53:02.349890 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:53:02.349855 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c72f864_73b2_48b6_ba6c_fe16323a2d18.slice/crio-480b39b91d71a81867488ab2dcb39795bf944e522ca2ad76f0b2934d3e0b9bab WatchSource:0}: Error finding container 480b39b91d71a81867488ab2dcb39795bf944e522ca2ad76f0b2934d3e0b9bab: Status 404 returned error can't find the container with id 480b39b91d71a81867488ab2dcb39795bf944e522ca2ad76f0b2934d3e0b9bab Oct 30 05:53:02.392738 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.392666 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr"] Oct 30 05:53:02.396826 ip-10-0-128-31 kubenswrapper[2578]: W1030 05:53:02.396790 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52fc9a2_fabd_41a9_8626_6a2aae3bf4b9.slice/crio-fa33d9537fe0d419cfbf78e5cd7951c0483c7e9e559badf6d2361a002a545a8c WatchSource:0}: Error finding container fa33d9537fe0d419cfbf78e5cd7951c0483c7e9e559badf6d2361a002a545a8c: Status 404 returned error can't find the container with id fa33d9537fe0d419cfbf78e5cd7951c0483c7e9e559badf6d2361a002a545a8c Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.759412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.759495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.759540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.759558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.759576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759589 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759617 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z: secret "image-registry-tls" not found Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.759623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759663 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759663 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759684 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls podName:184d6270-19ed-4e6b-80c6-9734584b82a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.759663035 +0000 UTC m=+45.669396251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls") pod "image-registry-7f9d7c8b8f-gxn9z" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1") : secret "image-registry-tls" not found Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759713 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls podName:800e718b-7419-4df9-836a-cea3686ceda9 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.759698489 +0000 UTC m=+45.669431690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls") pod "dns-default-5gv9v" (UID: "800e718b-7419-4df9-836a-cea3686ceda9") : secret "dns-default-metrics-tls" not found Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759716 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759730 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert podName:d6d3694c-d99c-43d1-be3f-6df16717e1de nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.759720633 +0000 UTC m=+45.669453834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5twq8" (UID: "d6d3694c-d99c-43d1-be3f-6df16717e1de") : secret "networking-console-plugin-cert" not found Oct 30 05:53:02.759983 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759751 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.759740153 +0000 UTC m=+45.669473384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : secret "router-metrics-certs-default" not found Oct 30 05:53:02.760741 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759774 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.759758529 +0000 UTC m=+45.669491731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : configmap references non-existent config key: service-ca.crt Oct 30 05:53:02.760741 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759775 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 05:53:02.760741 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.759663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:02.760741 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759838 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:53:02.760741 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759849 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9c9cf4b4-89zvd: secret "image-registry-tls" not found Oct 30 05:53:02.760741 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.759854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:53:02.760741 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759889 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls podName:00361e44-6e15-460e-88e7-1fd0398406e5 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.75987886 +0000 UTC m=+45.669612079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls") pod "image-registry-7b9c9cf4b4-89zvd" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5") : secret "image-registry-tls" not found Oct 30 05:53:02.760741 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759908 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls podName:42155cb9-5b97-4aa0-8a18-70af60804e34 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.759900287 +0000 UTC m=+45.669633486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkktj" (UID: "42155cb9-5b97-4aa0-8a18-70af60804e34") : secret "cluster-monitoring-operator-tls" not found Oct 30 05:53:02.760741 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759923 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:53:02.760741 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:02.759958 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls podName:6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.759947898 +0000 UTC m=+45.669681099 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-glfjr" (UID: "6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef") : secret "samples-operator-tls" not found Oct 30 05:53:02.953167 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.953128 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr" event={"ID":"f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9","Type":"ContainerStarted","Data":"fa33d9537fe0d419cfbf78e5cd7951c0483c7e9e559badf6d2361a002a545a8c"} Oct 30 05:53:02.955625 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.955567 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts" event={"ID":"6c72f864-73b2-48b6-ba6c-fe16323a2d18","Type":"ContainerStarted","Data":"480b39b91d71a81867488ab2dcb39795bf944e522ca2ad76f0b2934d3e0b9bab"} Oct 30 05:53:02.958705 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.958660 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" event={"ID":"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9","Type":"ContainerStarted","Data":"65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725"} Oct 30 05:53:02.958834 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:02.958711 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" event={"ID":"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9","Type":"ContainerStarted","Data":"7e40cc373f4722e46eeee669f2b7f968458827ab0e49b430efa82d3190db4463"} Oct 30 05:53:03.164384 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:03.164352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:03.164576 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:03.164493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:03.164679 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:03.164663 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:03.164737 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:03.164731 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:05.164711935 +0000 UTC m=+40.074445147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:03.165059 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:03.165040 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:53:03.165155 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:03.165094 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:05.165079609 +0000 UTC m=+40.074812826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : configmap "kube-rbac-proxy" not found Oct 30 05:53:03.964805 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:03.964765 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsrdq" event={"ID":"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8","Type":"ContainerStarted","Data":"a142247f38d4339b7811547699152dcbd605b8884d858d7a9d300c4c5b30f057"} Oct 30 05:53:04.007199 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:04.007145 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xsrdq" podStartSLOduration=35.972557719 podStartE2EDuration="39.007127741s" totalImagesPullingTime="3.034570022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:04.005500924 +0000 UTC m=+38.915234147" watchObservedRunningTime="2026-10-30 05:53:04.007127741 +0000 UTC m=+38.916860963" Oct 30 05:53:04.008667 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:04.008613 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" podStartSLOduration=3.008600487 podStartE2EDuration="3.008600487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:03.002030603 +0000 UTC m=+37.911763823" watchObservedRunningTime="2026-10-30 05:53:04.008600487 +0000 UTC m=+38.918333709" Oct 30 05:53:04.970459 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:04.970413 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr" event={"ID":"f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9","Type":"ContainerStarted","Data":"ebdaa317035b521797a03fc0797ac1ebc498931d7dc2a48936a9816dd2aca78e"} Oct 30 05:53:04.972078 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:04.972052 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts" event={"ID":"6c72f864-73b2-48b6-ba6c-fe16323a2d18","Type":"ContainerStarted","Data":"2fd8dfa87f84f180b3bfbe330c1e96c5317c1d09830dad8ca0fd2fe0207191d4"} Oct 30 05:53:04.972205 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:04.972181 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:53:05.011197 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:05.011143 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr" podStartSLOduration=2.324968269 podStartE2EDuration="4.01112554s" totalImagesPullingTime="1.686157271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:05.009063007 +0000 UTC m=+39.918796230" watchObservedRunningTime="2026-10-30 05:53:05.01112554 +0000 UTC m=+39.920858763" Oct 30 05:53:05.051670 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:05.051621 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts" podStartSLOduration=2.32061709 podStartE2EDuration="4.051606628s" totalImagesPullingTime="1.730989538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:05.051405004 +0000 UTC m=+39.961138226" watchObservedRunningTime="2026-10-30 05:53:05.051606628 +0000 UTC m=+39.961339848" Oct 30 05:53:05.174846 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:05.174805 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dz874_7212e5ac-eea5-438a-a162-74821cf38b76/dns-node-resolver/0.log" Oct 30 05:53:05.187228 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:05.187201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:05.187382 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:05.187323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:05.187382 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:05.187334 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:53:05.187502 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:05.187410 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:09.187391194 +0000 UTC m=+44.097124409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : configmap "kube-rbac-proxy" not found Oct 30 05:53:05.187502 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:05.187429 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:05.187502 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:05.187466 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:09.187456118 +0000 UTC m=+44.097189317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:05.284835 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:05.284758 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:05.284835 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:05.284825 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:05.285173 ip-10-0-128-31 kubenswrapper[2578]: I1030 05:53:05.285160 2578 scope.go:122] "RemoveContainer" containerID="a1b0e58f3452ccffe96b15479c7192f31726ad35f4c7c369ea791c73d61053fe" Oct 30 05:53:05.285397 ip-10-0-128-31 kubenswrapper[2578]: E1030 05:53:05.285378 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7ztmq_openshift-console-operator(cb9c2d4d-67e5-4c4d-8294-86ae9050232a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" podUID="cb9c2d4d-67e5-4c4d-8294-86ae9050232a" Oct 30 05:53:05.906934 ip-10-0-128-31 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 05:53:05.920688 ip-10-0-128-31 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 05:53:05.920896 ip-10-0-128-31 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 05:53:05.920957 ip-10-0-128-31 systemd[1]: kubelet.service: Consumed 3.077s CPU time. Oct 30 05:53:05.936894 ip-10-0-128-31 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:53:06.034703 ip-10-0-128-31 kubenswrapper[5001]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:53:06.034703 ip-10-0-128-31 kubenswrapper[5001]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:53:06.034703 ip-10-0-128-31 kubenswrapper[5001]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:53:06.034703 ip-10-0-128-31 kubenswrapper[5001]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:53:06.036779 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036694 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:06.036779 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036779 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036784 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036788 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036791 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036793 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036797 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036800 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036803 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036807 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036812 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036815 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036819 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036821 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036824 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036827 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036830 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036836 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036841 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036845 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:06.036855 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036849 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036853 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036856 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036861 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036865 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036868 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036871 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036874 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036876 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036878 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036881 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036884 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036886 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036889 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036891 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036894 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036896 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036899 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036901 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036903 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:06.037372 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036906 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036908 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036911 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036913 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036915 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036918 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036921 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036925 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036928 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036930 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036933 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036936 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036939 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036941 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036944 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036946 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036949 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036951 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036954 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036956 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:06.037852 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036959 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036961 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036963 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036966 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036969 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036971 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036973 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036976 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036978 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036981 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036983 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036987 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036991 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036994 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036996 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.036999 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037001 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037004 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037007 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:06.038355 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037009 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037013 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037016 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037018 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037021 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037023 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037026 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037028 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037031 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037033 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037036 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037038 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037041 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037044 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037048 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037050 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037053 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037055 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037057 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:06.038798 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037060 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037062 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037065 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037068 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037070 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037072 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037075 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037077 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037885 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037893 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037896 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037899 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037902 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037905 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037907 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037911 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037914 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037916 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037919 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037922 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:06.039252 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037928 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037930 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037933 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037936 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037939 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037942 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037944 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037947 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037949 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037952 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037954 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037957 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037960 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037962 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037965 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037968 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037970 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037973 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037975 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037978 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:06.039760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037980 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037983 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037985 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037987 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037993 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037995 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.037998 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038000 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038003 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038006 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038008 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038011 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038013 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038015 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038018 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038021 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038023 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038026 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038028 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:06.040240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038030 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038033 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038035 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038038 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038041 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038044 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038047 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038049 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038052 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038054 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038057 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038059 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038062 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038064 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038067 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038069 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038072 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038075 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038078 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:06.040716 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038080 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038083 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038085 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038090 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038094 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038097 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038100 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038103 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038106 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038110 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038112 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038115 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038117 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038120 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038123 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038125 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038128 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038131 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038133 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:06.041169 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038137 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038140 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038142 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038144 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038147 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038149 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038152 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038154 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038157 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038159 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038161 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038165 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038167 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038170 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038172 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038175 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038177 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038695 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038702 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038706 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:06.041676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038709 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038712 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038714 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038717 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038719 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038722 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038725 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038727 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038730 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038732 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038735 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038738 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038741 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038743 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038746 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038748 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038751 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038753 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038755 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038758 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:06.042150 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038760 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038763 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038765 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038768 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038771 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038774 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038792 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038797 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038800 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038807 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038810 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038813 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038816 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038818 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038820 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038823 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038825 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038828 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038831 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:06.042663 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038834 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038838 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038841 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038844 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038847 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038849 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038852 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038854 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038857 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038861 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038863 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038866 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038868 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038871 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038873 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038876 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038878 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038881 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038883 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038885 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:06.043152 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038888 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038890 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038893 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038896 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038898 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038900 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038903 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038905 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038907 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038910 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038912 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038915 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038917 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038920 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038922 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038924 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038927 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038929 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038931 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038937 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:06.043666 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038940 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038942 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038945 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038947 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038950 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038952 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038955 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038957 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038960 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038963 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038965 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038967 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038970 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038972 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038976 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038980 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038984 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038987 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038989 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:06.044186 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038992 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038994 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038997 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.038999 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039002 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039098 5001 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039117 5001 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039126 5001 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039133 5001 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039138 5001 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039141 5001 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039145 5001 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039156 5001 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039159 5001 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039162 5001 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039166 5001 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039169 5001 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039172 5001 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039175 5001 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039177 5001 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039180 5001 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039183 5001 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039186 5001 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:53:06.044643 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039190 5001 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039193 5001 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039196 5001 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039199 5001 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039203 5001 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039207 5001 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039209 5001 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039213 5001 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039217 5001 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039220 5001 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039223 5001 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039225 5001 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039228 5001 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039231 5001 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039236 5001 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039239 5001 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039242 5001 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039245 5001 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039248 5001 flags.go:64] FLAG: --enable-server="true" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039250 5001 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039255 5001 flags.go:64] FLAG: --event-burst="100" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039260 5001 flags.go:64] FLAG: --event-qps="50" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039265 5001 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039267 5001 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:53:06.045201 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039270 5001 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039275 5001 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039278 5001 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039281 5001 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039284 5001 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039287 5001 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039290 5001 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039293 5001 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039295 5001 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039298 5001 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039317 5001 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039321 5001 flags.go:64] FLAG: --feature-gates="" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039326 5001 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039329 5001 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039332 5001 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039335 5001 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039339 5001 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039342 5001 flags.go:64] FLAG: --help="false" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039345 5001 flags.go:64] FLAG: --hostname-override="ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039348 5001 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039351 5001 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039353 5001 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039357 5001 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:53:06.045788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039360 5001 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039363 5001 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039366 5001 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039369 5001 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039372 5001 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039375 5001 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039379 5001 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039383 5001 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039386 5001 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039389 5001 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039392 5001 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039394 5001 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039397 5001 flags.go:64] FLAG: --lock-file="" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039400 5001 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039403 5001 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039406 5001 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039411 5001 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039414 5001 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039417 5001 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039419 5001 flags.go:64] FLAG: --logging-format="text" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039422 5001 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039426 5001 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039428 5001 flags.go:64] FLAG: --manifest-url="" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039431 5001 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039435 5001 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:53:06.046364 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039439 5001 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039443 5001 flags.go:64] FLAG: --max-pods="110" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039446 5001 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039449 5001 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039452 5001 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039455 5001 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039458 5001 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039461 5001 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039464 5001 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039470 5001 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039473 5001 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039475 5001 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039478 5001 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039482 5001 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039486 5001 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039489 5001 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039492 5001 flags.go:64] FLAG: --port="10250" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039495 5001 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039498 5001 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01051a8400d4b8709" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039501 5001 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039504 5001 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039507 5001 flags.go:64] FLAG: --register-node="true" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039509 5001 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039513 5001 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039516 5001 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039519 5001 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:53:06.046970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039522 5001 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039526 5001 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039529 5001 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039532 5001 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039535 5001 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039537 5001 flags.go:64] FLAG: --runonce="false" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039540 5001 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039543 5001 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039547 5001 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039550 5001 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039552 5001 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039555 5001 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039558 5001 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039561 5001 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039564 5001 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039567 5001 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039570 5001 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039573 5001 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039575 5001 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039579 5001 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039584 5001 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039589 5001 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039592 5001 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039594 5001 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039598 5001 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:53:06.047678 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039601 5001 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039603 5001 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039606 5001 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039609 5001 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039612 5001 flags.go:64] FLAG: --v="2" Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039616 5001 flags.go:64] FLAG: --version="false" Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039620 5001 flags.go:64] FLAG: --vmodule="" Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039624 5001 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.039627 5001 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039757 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039761 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039764 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039767 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039770 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039773 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039775 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039778 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039781 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039783 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039786 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039789 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:06.048741 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039791 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039793 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039796 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039798 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039801 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039804 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039808 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039810 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039813 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039816 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039818 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039821 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039823 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039826 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039828 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039831 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039833 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039836 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039838 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039841 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:06.049505 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039843 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039845 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039848 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039850 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039853 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039864 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039867 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039870 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039873 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039875 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039878 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039880 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039883 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039885 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039888 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039891 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039894 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039900 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039905 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:06.050376 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039907 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039910 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039913 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039915 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039918 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039920 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039923 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039925 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039928 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039930 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039933 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039935 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039938 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039940 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039943 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039945 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039948 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039950 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039953 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039955 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:06.050926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039958 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039960 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039963 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039966 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039969 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039971 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039974 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039976 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039978 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039981 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039984 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039988 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039990 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039993 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039995 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.039998 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040000 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040003 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040005 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040009 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:06.051544 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040013 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040017 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040019 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040022 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040024 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040027 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040030 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040032 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040034 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040037 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040040 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040042 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040044 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040047 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040049 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:06.052219 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.040075 5001 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040122 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040126 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040129 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040132 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040134 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040137 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040139 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040145 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040149 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040153 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040156 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040159 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040161 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040164 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040166 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040169 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040172 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040174 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:06.052926 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040177 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040179 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040182 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040184 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040187 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040189 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040192 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040194 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040197 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040200 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040202 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040204 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040207 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040209 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040212 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040215 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040217 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040219 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040222 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:06.053731 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040224 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040227 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040231 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040234 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040236 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040239 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040241 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040244 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040247 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040250 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040252 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040255 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040257 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040260 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040262 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040265 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040267 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040269 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040272 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:06.054417 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040274 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040277 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040279 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040281 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040284 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040286 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040289 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040291 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040293 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040296 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040299 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040318 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040321 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040323 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040326 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040331 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040335 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040340 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040342 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:06.054949 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040345 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040347 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040350 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040352 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040355 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040357 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040360 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040362 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040365 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040368 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040371 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040373 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040376 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040378 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040381 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040383 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040386 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040388 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040391 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040394 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:06.055567 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040396 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040398 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040401 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040404 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040407 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040409 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040412 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040414 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040419 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040421 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.040424 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.044529 5001 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.044543 5001 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.044561 5001 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.044566 5001 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044620 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044625 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044629 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044632 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:06.056047 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044636 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044638 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044641 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044644 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044647 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044649 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044652 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044655 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044658 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044661 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044664 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044666 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044669 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044671 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044674 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044677 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044680 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044683 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044686 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:06.056525 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044688 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044691 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044694 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044697 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044699 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044702 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044704 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044708 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044711 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044714 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044717 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044720 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044722 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044725 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044727 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044730 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044732 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044735 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044737 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:06.056984 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044740 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044743 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044745 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044748 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044751 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044753 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044756 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044758 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044761 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044764 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044766 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044769 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044772 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044775 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044777 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044781 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044783 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044786 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044788 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044791 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:06.057490 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044793 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044796 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044798 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044801 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044803 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044806 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044808 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044810 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044813 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044815 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044818 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044820 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044823 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044825 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044827 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044830 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044832 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044834 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044837 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044839 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:06.057972 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044842 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044845 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044847 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044849 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044852 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044854 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044857 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044859 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044862 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044865 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044869 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044873 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044876 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044878 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044881 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044883 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044885 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044888 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044890 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044892 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:06.058471 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044895 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044897 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044900 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.044902 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.044929 5001 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045056 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045060 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045063 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045066 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045068 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045071 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045073 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045076 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045078 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045081 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045083 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:06.058938 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045085 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045092 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045095 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045097 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045100 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045102 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045104 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045107 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045110 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045112 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045114 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045117 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045119 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045123 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045126 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045130 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045132 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045135 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045138 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:06.059386 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045140 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045143 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045145 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045148 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045150 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045153 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045155 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045158 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045161 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045163 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045167 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045170 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045173 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045175 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045178 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045181 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045183 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045186 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045188 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045190 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:06.059833 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045193 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045195 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045198 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045200 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045203 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045206 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045208 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045211 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045214 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045216 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045219 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045221 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045224 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045226 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045228 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045231 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045233 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045236 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045238 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:06.060341 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045241 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045243 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045246 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045248 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045250 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045253 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045255 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045258 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045260 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045263 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045265 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045268 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045270 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045273 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045275 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045277 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045280 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045282 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045284 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045287 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:06.060850 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045290 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045292 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045295 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045297 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045299 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045325 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045328 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045330 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045333 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045335 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045337 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045340 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045342 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045345 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045347 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045350 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:06.061365 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045352 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.045376 5001 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045406 5001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045410 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045413 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045416 5001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045419 5001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045421 5001 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045424 5001 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045426 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045429 5001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045431 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045434 5001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045436 5001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045440 5001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045443 5001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:06.061760 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045447 5001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045449 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045452 5001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045454 5001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045457 5001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045459 5001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045461 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045464 5001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045466 5001 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045468 5001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045471 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045473 5001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045475 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045478 5001 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045481 5001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045483 5001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045485 5001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045488 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045494 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045496 5001 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:06.062143 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045499 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045502 5001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045504 5001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045507 5001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045509 5001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045512 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045514 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045517 5001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045519 5001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045522 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045524 5001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045526 5001 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045529 5001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045531 5001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045534 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045536 5001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045538 5001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045542 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045544 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045546 5001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:06.062676 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045549 5001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045551 5001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045553 5001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045556 5001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045558 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045560 5001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045563 5001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045566 5001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045569 5001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045571 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045575 5001 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045578 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045580 5001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045583 5001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045586 5001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045588 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045592 5001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045594 5001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045597 5001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:06.063157 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045599 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045602 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045604 5001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045606 5001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045609 5001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045611 5001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045614 5001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045616 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045619 5001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045621 5001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045623 5001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045626 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045628 5001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045631 5001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045633 5001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045635 5001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045637 5001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045640 5001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045644 5001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045646 5001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:06.063733 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045649 5001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045651 5001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045653 5001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045657 5001 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045659 5001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045661 5001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045664 5001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045666 5001 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045669 5001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045673 5001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045678 5001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045681 5001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:06.045684 5001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.045730 5001 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.046672 5001 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.046728 5001 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.046903 5001 server.go:1014] "Starting client certificate rotation" Oct 30 05:53:06.064222 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.047027 5001 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:53:06.064659 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.047072 5001 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:47:25 +0000 UTC" deadline="2028-06-27 11:25:14.840431336 +0000 UTC" Oct 30 05:53:06.064659 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.047098 5001 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14549h32m8.79333521s" Oct 30 05:53:06.064659 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.047336 5001 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:53:06.064659 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.049651 5001 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:53:06.064659 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.051484 5001 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:53:06.064659 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.053616 5001 log.go:25] "Validated CRI v1 image API" Oct 30 05:53:06.064659 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.054423 5001 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:53:06.064659 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.056385 5001 fs.go:135] Filesystem UUIDs: map[4bfed4fc-9bbf-4f82-a1a5-899e02372e24:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d525d619-e4c4-4133-a33e-9afb59a77725:/dev/nvme0n1p4] Oct 30 05:53:06.064885 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.056401 5001 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e40d083151df3cf253eb758bac5c948afc8977d7cacbd1beb526d8258094ee1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e40d083151df3cf253eb758bac5c948afc8977d7cacbd1beb526d8258094ee1/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dec71dde8963b69ee2b5f03cd1adb9d18ccc4d80f64dd47757147179b57f692/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dec71dde8963b69ee2b5f03cd1adb9d18ccc4d80f64dd47757147179b57f692/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/303f050fdbf7b3c43f865d3732ffc24453c1c8fcdc7146738e23217094f5a4f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/303f050fdbf7b3c43f865d3732ffc24453c1c8fcdc7146738e23217094f5a4f4/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3979a4dbf45de6174e65b75182d05294cd60f1fab756ae5cec10d8f469f33e21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3979a4dbf45de6174e65b75182d05294cd60f1fab756ae5cec10d8f469f33e21/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/480b39b91d71a81867488ab2dcb39795bf944e522ca2ad76f0b2934d3e0b9bab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/480b39b91d71a81867488ab2dcb39795bf944e522ca2ad76f0b2934d3e0b9bab/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c453175eff516c698e66a1b7fca3d1cc5b61ff55636b4bf3f2082ccaa4ff1a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c453175eff516c698e66a1b7fca3d1cc5b61ff55636b4bf3f2082ccaa4ff1a9/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50fe92da315b6dc10e5250bca9dcb85546c6e5aa696bf1c3dff174ef9e1bc17e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50fe92da315b6dc10e5250bca9dcb85546c6e5aa696bf1c3dff174ef9e1bc17e/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5264bed43ef0b66623dec4796da7b3bb76bfbb97bffd3026398a3dccb6a16e40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5264bed43ef0b66623dec4796da7b3bb76bfbb97bffd3026398a3dccb6a16e40/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a56192fcd20a9af9aad204e5d2162d757e993b589a9e3908b4188cb024d17e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a56192fcd20a9af9aad204e5d2162d757e993b589a9e3908b4188cb024d17e8/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cc4e16f243d0dbbf7104b96dc1cfb68747d1ea324b3f55e870ac53b7664d209/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cc4e16f243d0dbbf7104b96dc1cfb68747d1ea324b3f55e870ac53b7664d209/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5fa96e71655df633bb5f03b03d39f54c418e1841d26a7b50e96f240648c751f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5fa96e71655df633bb5f03b03d39f54c418e1841d26a7b50e96f240648c751f5/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6167377e81c1d4b4f582f023dc700b09a8340d50657f21a7b7a131a41dccd9f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6167377e81c1d4b4f582f023dc700b09a8340d50657f21a7b7a131a41dccd9f3/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7951402aeba57244ffbe2add8b3ef385c13e12c15f4417740d8c6a7b3e06f75d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7951402aeba57244ffbe2add8b3ef385c13e12c15f4417740d8c6a7b3e06f75d/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e40cc373f4722e46eeee669f2b7f968458827ab0e49b430efa82d3190db4463/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e40cc373f4722e46eeee669f2b7f968458827ab0e49b430efa82d3190db4463/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/937c4a9525ad300b9baf2e749750324f985f733829dcf4fe937199dd621e013c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/937c4a9525ad300b9baf2e749750324f985f733829dcf4fe937199dd621e013c/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98a66792bd4248f2fdec8187af68dddc5fd7e1057d997408076f5b40c902e33c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98a66792bd4248f2fdec8187af68dddc5fd7e1057d997408076f5b40c902e33c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a10d6ef400c2aaada873e76b1aba180329a6c526c265c296884109e30e9dbbe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a10d6ef400c2aaada873e76b1aba180329a6c526c265c296884109e30e9dbbe/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ab94d7e66d56937833d15fcb51b0c16944de28897faecdfd3d843d6aa885ca8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ab94d7e66d56937833d15fcb51b0c16944de28897faecdfd3d843d6aa885ca8/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7c4fcdeb47e184e4c5305e69352178adb81dc48402fd7b3a6fb694cd0ee0697/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7c4fcdeb47e184e4c5305e69352178adb81dc48402fd7b3a6fb694cd0ee0697/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7e8d3e8247283e32225098487ec34520a2a90cb84181d0b971e199a64baab0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7e8d3e8247283e32225098487ec34520a2a90cb84181d0b971e199a64baab0e/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0fd2dfc7fa02952cd90a71420c4870c6f0e1fc33234dec4eac740c3c937504a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0fd2dfc7fa02952cd90a71420c4870c6f0e1fc33234dec4eac740c3c937504a/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8db7439dddd8b2f53cd4e6ac413734c8079dfd5e6267ba1170fd427879760fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8db7439dddd8b2f53cd4e6ac413734c8079dfd5e6267ba1170fd427879760fc/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4dbbaf4e2c1247524a00d3c23a0803587d49d72584cd287c53b4e9200982013/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4dbbaf4e2c1247524a00d3c23a0803587d49d72584cd287c53b4e9200982013/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f941e4e4b67f967090a4170d32b43b293f91cf56a16c0bf5af4ec29cb49c2613/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f941e4e4b67f967090a4170d32b43b293f91cf56a16c0bf5af4ec29cb49c2613/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa33d9537fe0d419cfbf78e5cd7951c0483c7e9e559badf6d2361a002a545a8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa33d9537fe0d419cfbf78e5cd7951c0483c7e9e559badf6d2361a002a545a8c/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~projected/kube-api-access-fhp65:{mountpoint:/var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~projected/kube-api-access-fhp65 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02b629c0-be01-461a-88b8-85a69dfad05b/volumes/kubernetes.io~projected/kube-api-access-lmlgr:{mountpoint:/var/lib/kubelet/pods/02b629c0-be01-461a-88b8-85a69dfad05b/volumes/kubernetes.io~projected/kube-api-access-lmlgr major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11393f29-8fd7-4a36-b3df-03f0992fc7ad/volumes/kubernetes.io~projected/kube-api-access-wx5hv:{mountpoint:/var/lib/kubelet/pods/11393f29-8fd7-4a36-b3df-03f0992fc7ad/volumes/kubernetes.io~projected/kube-api-access-wx5hv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~projected/kube-api-access-krbp7:{mountpoint:/var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~projected/kube-api-access-krbp7 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b97e5c9-769c-4c97-8847-50a1d8ad613d/volumes/kubernetes.io~projected/kube-api-access-2ngjf:{mountpoint:/var/lib/kubelet/pods/1b97e5c9-769c-4c97-8847-50a1d8ad613d/volumes/kubernetes.io~projected/kube-api-access-2ngjf major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391996f2-6666-4d62-9af2-1291a66828bf/volumes/kubernetes.io~projected/kube-api-access-8srpr:{mountpoint:/var/lib/kubelet/pods/391996f2-6666-4d62-9af2-1291a66828bf/volumes/kubernetes.io~projected/kube-api-access-8srpr major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391996f2-6666-4d62-9af2-1291a66828bf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/391996f2-6666-4d62-9af2-1291a66828bf/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b6113ea-d183-4a72-a574-80c03c8f5352/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3b6113ea-d183-4a72-a574-80c03c8f5352/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42155cb9-5b97-4aa0-8a18-70af60804e34/volumes/kubernetes.io~projected/kube-api-access-l52zw:{mountpoint:/var/lib/kubelet/pods/42155cb9-5b97-4aa0-8a18-70af60804e34/volumes/kubernetes.io~projected/kube-api-access-l52zw major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9/volumes/kubernetes.io~projected/kube-api-access-hgl2n:{mountpoint:/var/lib/kubelet/pods/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9/volumes/kubernetes.io~projected/kube-api-access-hgl2n major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8/volumes/kubernetes.io~projected/kube-api-access-d2xjq:{mountpoint:/var/lib/kubelet/pods/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8/volumes/kubernetes.io~projected/kube-api-access-d2xjq major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67fb2af9-df3c-41a8-997c-e16de0871a3e/volumes/kubernetes.io~projected/kube-api-access-wdslj:{mountpoint:/var/lib/kubelet/pods/67fb2af9-df3c-41a8-997c-e16de0871a3e/volumes/kubernetes.io~projected/kube-api-access-wdslj major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67fb2af9-df3c-41a8-997c-e16de0871a3e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/67fb2af9-df3c-41a8-997c-e16de0871a3e/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef/volumes/kubernetes.io~projected/kube-api-access-wn9vt:{mountpoint:/var/lib/kubelet/pods/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef/volumes/kubernetes.io~projected/kube-api-access-wn9vt major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c72f864-73b2-48b6-ba6c-fe16323a2d18/volumes/kubernetes.io~projected/kube-api-access-nhgdv:{mountpoint:/var/lib/kubelet/pods/6c72f864-73b2-48b6-ba6c-fe16323a2d18/volumes/kubernetes.io~projected/kube-api-access-nhgdv major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7212e5ac-eea5-438a-a162-74821cf38b76/volumes/kubernetes.io~projected/kube-api-access-mhkl4:{mountpoint:/var/lib/kubelet/pods/7212e5ac-eea5-438a-a162-74821cf38b76/volumes/kubernetes.io~projected/kube-api-access-mhkl4 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/800e718b-7419-4df9-836a-cea3686ceda9/volumes/kubernetes.io~projected/kube-api-access-msw47:{mountpoint:/var/lib/kubelet/pods/800e718b-7419-4df9-836a-cea3686ceda9/volumes/kubernetes.io~projected/kube-api-access-msw47 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84346da8-55f2-4344-b7d0-dbfb2996f2e7/volumes/kubernetes.io~projected/kube-api-access-szdrr:{mountpoint:/var/lib/kubelet/pods/84346da8-55f2-4344-b7d0-dbfb2996f2e7/volumes/kubernetes.io~projected/kube-api-access-szdrr major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84346da8-55f2-4344-b7d0-dbfb2996f2e7/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/84346da8-55f2-4344-b7d0-dbfb2996f2e7/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84346da8-55f2-4344-b7d0-dbfb2996f2e7/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/84346da8-55f2-4344-b7d0-dbfb2996f2e7/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a5d5930-3046-4003-bce1-4716ab70ef86/volumes/kubernetes.io~projected/kube-api-access-9z9x5:{mountpoint:/var/lib/kubelet/pods/8a5d5930-3046-4003-bce1-4716ab70ef86/volumes/kubernetes.io~projected/kube-api-access-9z9x5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cd01f41-97de-437c-aacf-29cb3668570b/volumes/kubernetes.io~projected/kube-api-access-8rx4n:{mountpoint:/var/lib/kubelet/pods/8cd01f41-97de-437c-aacf-29cb3668570b/volumes/kubernetes.io~projected/kube-api-access-8rx4n major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f29d013-5eaf-42aa-860a-ba18193d7807/volumes/kubernetes.io~projected/kube-api-access-sfpbt:{mountpoint:/var/lib/kubelet/pods/8f29d013-5eaf-42aa-860a-ba18193d7807/volumes/kubernetes.io~projected/kube-api-access-sfpbt major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f29d013-5eaf-42aa-860a-ba18193d7807/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8f29d013-5eaf-42aa-860a-ba18193d7807/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93ed3e75-7873-4300-be92-0cff9a0d63a7/volumes/kubernetes.io~projected/kube-api-access-z7k4q:{mountpoint:/var/lib/kubelet/pods/93ed3e75-7873-4300-be92-0cff9a0d63a7/volumes/kubernetes.io~projected/kube-api-access-z7k4q major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93ed3e75-7873-4300-be92-0cff9a0d63a7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/93ed3e75-7873-4300-be92-0cff9a0d63a7/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a792a513-44ef-43d1-a48c-e17d8f939042/volumes/kubernetes.io~projected/kube-api-access-lmfq7:{mountpoint:/var/lib/kubelet/pods/a792a513-44ef-43d1-a48c-e17d8f939042/volumes/kubernetes.io~projected/kube-api-access-lmfq7 major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac9c2fad-cc58-4ec5-b169-271b6937af4b/volumes/kubernetes.io~projected/kube-api-access-g8ltg:{mountpoint:/var/lib/kubelet/pods/ac9c2fad-cc58-4ec5-b169-271b6937af4b/volumes/kubernetes.io~projected/kube-api-access-g8ltg major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5e82585-9fee-4f52-a796-ef82b220e6f6/volumes/kubernetes.io~projected/kube-api-access-95mdj:{mountpoint:/var/lib/kubelet/pods/b5e82585-9fee-4f52-a796-ef82b220e6f6/volumes/kubernetes.io~projected/kube-api-access-95mdj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7e5b861-eb47-4d8a-8974-90a2368489ed/volumes/kubernetes.io~projected/kube-api-access-p6tct:{mountpoint:/var/lib/kubelet/pods/c7e5b861-eb47-4d8a-8974-90a2368489ed/volumes/kubernetes.io~projected/kube-api-access-p6tct major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cad7f8eb-8604-4b8b-a921-38b9d5c281db/volumes/kubernetes.io~projected/kube-api-access-prjhg:{mountpoint:/var/lib/kubelet/pods/cad7f8eb-8604-4b8b-a921-38b9d5c281db/volumes/kubernetes.io~projected/kube-api-access-prjhg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb9c2d4d-67e5-4c4d-8294-86ae9050232a/volumes/kubernetes.io~projected/kube-api-access-vzpmr:{mountpoint:/var/lib/kubelet/pods/cb9c2d4d-67e5-4c4d-8294-86ae9050232a/volumes/kubernetes.io~projected/kube-api-access-vzpmr major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb9c2d4d-67e5-4c4d-8294-86ae9050232a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cb9c2d4d-67e5-4c4d-8294-86ae9050232a/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d44df296-4186-4c40-81de-1ed06c15955e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d44df296-4186-4c40-81de-1ed06c15955e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de6988ac-1b54-4229-9424-aa8e28cbb6c6/volumes/kubernetes.io~projected/kube-api-access-nzp62:{mountpoint:/var/lib/kubelet/pods/de6988ac-1b54-4229-9424-aa8e28cbb6c6/volumes/kubernetes.io~projected/kube-api-access-nzp62 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1d67bea-af2e-44ad-ab9b-01123c2889fc/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f1d67bea-af2e-44ad-ab9b-01123c2889fc/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1d67bea-af2e-44ad-ab9b-01123c2889fc/volumes/kubernetes.io~projected/kube-api-access-rznb5:{mountpoint:/var/lib/kubelet/pods/f1d67bea-af2e-44ad-ab9b-01123c2889fc/volumes/kubernetes.io~projected/kube-api-access-rznb5 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1d67bea-af2e-44ad-ab9b-01123c2889fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f1d67bea-af2e-44ad-ab9b-01123c2889fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9/volumes/kubernetes.io~projected/kube-api-access-ftrml:{mountpoint:/var/lib/kubelet/pods/f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9/volumes/kubernetes.io~projected/kube-api-access-ftrml major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7ce5e27-0a29-46db-a45d-94dd98cbe059/volumes/kubernetes.io~projected/kube-api-access-cxjtk:{mountpoint:/var/lib/kubelet/pods/f7ce5e27-0a29-46db-a45d-94dd98cbe059/volumes/kubernetes.io~projected/kube-api-access-cxjtk major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d/volumes/kubernetes.io~projected/kube-api-access-28fpx:{mountpoint:/var/lib/kubelet/pods/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d/volumes/kubernetes.io~projected/kube-api-access-28fpx major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a17318c3d2a1b2aa72f5d97b97ae8ea19ac3278d177d857f832092abdbb9ce24/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/50600c3ebf25299461ad347468a450b72817cccc47f62c24f50377cf067de95d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/081ed81a82b1809475591a66e990a9653a955c46e4d8b811f4ddc5c43df9efca/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/19e4dce75529e6fa374c144492296ce9996c543ccba5adaa8d83e377bdfe96e4/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a8a50dd40769fac7018ebeb3940705b5a2e1738d7e868cb584c78082a00662b8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b7303ff241cafc88bba1358b4e7dcd801ed95d31a969418acb930edcde4cc7aa/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/118196c5fc331bd64d33e8e01f89f54fdc82f7f8c66b5b90b28ce9cf2b80226d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/db1ffb16b8d237496811d94833c5adbbe8732b4fde5a2b8c76abe5728e852223/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ab1bcac84cef3ada6e9f139f4aa6d10c35c22fcba2125a73267978c2ad174dae/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/c8b7cec182c31eef02c46908f1afeb5c7bcb73512285564c92fb1def7d2cbf2c/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/c20306f693159570e5115adcf12159b0bbab5dfb7a95152954f0799930ad7017/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/35af0b2db19b63208141f701497be6a7d48f1f66d25b79178759a659cddd9e26/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/44090e1588a09a63f880818f312d2a8efd7874154e74498e6a483e3143cd39d1/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/08f21612b6ca238af139dd1589c458f4b2144d3ebe220bdea65226fd0e6fc317/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/da28fafb960b08a6ebfd2167b95c68aaa76645d712d8ca424d7d13e55bc9d736/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/98dbf9c023d902ddd60d448e9c6ab7b3f9cc4f914359d01714bf001258d2a06e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5039b032a113d2a7ed1366803ddc2979b1dd51160514ccfce504cf97695fff86/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/558a67397dabc6d13ba5907fb154c33b5681b8eb934a2e7e7454645405ca7579/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b3e709287e2a59a6044eaf7bc21ddad65a07d1949b9e799ef09ae48372c11bbb/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d975de6499a93e3b1768fe67820486ec88a1ed18de9d47fe935227a66c0f3bd8/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d37fc505964ffe1b8d31d54af6027aac458da2e082b209e22408c9a742529fad/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/26afbe51dc930596ce6acff822aa7380524371562afb19801667875188227b31/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/2039a2e610d663db80f79926f2bc6f92343ca09c5955ec345e67d73c31e2c7b3/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/ab3f75029139f6fe665df72ecd4aa99680adf8c18755c21b8ea8b18c0d9e141e/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/c0cfae8e3fff3dad0fa47886282abd8c9d70ed54bafffa456394901bf6c303f0/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/244620c0b0fa7c5823773dc0dd8aa54d814eeddeac0344c81166baecbf172c04/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/fc8c954d3d1cee0a834910b49f5870259c68c19767d517cd8efc833a3ccb6875/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/39c88d7abc2ce2d7d8b73dddd0468e98cefb514ab26c0d0fec38cbf3bff1c177/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d5f80896694d0c37e258e3a848c8a9362750656fb03f354c8740066cf2726fcb/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/2e622629d1c99682be7e0fcb8eb7aa9aaaff91cd746f96c14a9766565453be95/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/d6f11c857313b32744b1a25bc22f5b2fe68284670cd0f92b77b11b7ec2ddc164/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/2fc0ba6d33e73496576448bef17af81f06c3ccb0fa0ce1cb61cbe4a716c03813/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/f4506547644abc0a1e064692f977549a0b50a299507268b36b5a79161b470508/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/02f4b091f619295851b10fc1708f8d2ab78c6c83cc9b95ae1462bc8040cdac8b/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/f96a5dcf2e4191045e3c2c568eb46b96fb2b03cba21c991357825c91017310be/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/9e72f8004acee5820f2be2587fc7c55cb61bd98ccb48c78b6706a65b15d4da27/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/eee23122e5e3509fe5ad5b2d97b542dcc8943ded55d3879061162d31c1831d68/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/acca4a5997c30e13acf1f60d195bdbf6d91e3201da20ea9ad38efc7520a68286/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/7e79f163120617dc6cc9d634d23a7add437c53ff2bfe3d309a77e47412597e7b/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/124feca4862f8ccf330d2f950a0d3c09cb64c6aafe04dd6491b55025655ff63f/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/9345551922e771dc59a02d8218e5a12af6e61dfe42bb2f4be9af233679c90bca/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/c5b721d6759ae6bd35002e2cee061edfe6020e1821d9e3c2022503ee0eda2c40/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/9630e7b52c4098dde8219313709ba94a6edd6073d7e3b813d9d32ff3b4a8c0a3/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/b17c33ab68e4aa69ab4606848f239d89bc9288fb43d9657279a9721ac4f873b0/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/ef53dd6b21c892711c3a8605490a0f691a6a1eb9d832c8472bd025bca8cb3750/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/6b613e067ac1a5ac6320877f26a0998c7243df3182b53a03c75a8dad337130a0/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/fb148416940635a39018fdbd7a281b59928c86558ff672f7c4259dcf7fb95eb4/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/6665c4d6d6f311e419a76d4f883b471dbe80db02e7d78972173f1193e7d93a32/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/bdab05627282d1b9001b02603e37a7323077c3c8b43440a454c48bfe5562aed4/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/75be16835894954451dedf3c89a5c3e00d35476dd2232edfbbf4cfd38f59c984/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/541f60569b34139f7dd40b96ad8c06985b717a805306be999e752abfef56d13c/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9893d5a7a41e0d3188d29feaa77a4d60b389a6ae155569e0d55cae58d2be4a70/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/590a87e8ccc918daa4bc557dad36f5335a1a7bf9c925ea455c4c7940054e061c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/489319bbeb7215c7a01c104a55e09b0d737ae65c8d84c73a633bff06c11bda85/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3dddb8e558697d6185b3c7dceb14bba597aa193c0365e1c36483ee91d188e4d2/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c7048430268494e95ba1c3e9293054b875b469d45fe3d47258011fc6195bcbc7/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/51e7080d5fddf7e3080fc8a8bb569497ffc09ec2f184b04db6ff24b3341ce516/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/d62d03e413927204baf108359dad49aff79bc147d3d475c0a04ddbff1812f648/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ec3e5cf177609b6496c8093acddf07d7cefb76c6ebfb5b530cd38683c656b8d7/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/bd8194721461d5f54b4054bf896e0573305d4611ee89eb30be544845b79dce73/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 05:53:06.065087 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.064403 5001 manager.go:217] Machine: {Timestamp:2026-10-30 05:53:06.063932403 +0000 UTC m=+0.059889403 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093372 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26e662cb2e5378ea6af81a4ea81ff5 SystemUUID:ec26e662-cb2e-5378-ea6a-f81a4ea81ff5 BootID:879fd1ee-2deb-41d8-a7b0-a29c1bcc8fff Filesystems:[{Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84346da8-55f2-4344-b7d0-dbfb2996f2e7/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c7e5b861-eb47-4d8a-8974-90a2368489ed/volumes/kubernetes.io~projected/kube-api-access-p6tct DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e40d083151df3cf253eb758bac5c948afc8977d7cacbd1beb526d8258094ee1/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb9c2d4d-67e5-4c4d-8294-86ae9050232a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7c4fcdeb47e184e4c5305e69352178adb81dc48402fd7b3a6fb694cd0ee0697/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ab94d7e66d56937833d15fcb51b0c16944de28897faecdfd3d843d6aa885ca8/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d/volumes/kubernetes.io~projected/kube-api-access-28fpx DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84346da8-55f2-4344-b7d0-dbfb2996f2e7/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/391996f2-6666-4d62-9af2-1291a66828bf/volumes/kubernetes.io~projected/kube-api-access-8srpr DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9/volumes/kubernetes.io~projected/kube-api-access-hgl2n DeviceMajor:0 DeviceMinor:333 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa33d9537fe0d419cfbf78e5cd7951c0483c7e9e559badf6d2361a002a545a8c/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~projected/kube-api-access-krbp7 DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb9c2d4d-67e5-4c4d-8294-86ae9050232a/volumes/kubernetes.io~projected/kube-api-access-vzpmr DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d44df296-4186-4c40-81de-1ed06c15955e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b6113ea-d183-4a72-a574-80c03c8f5352/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a792a513-44ef-43d1-a48c-e17d8f939042/volumes/kubernetes.io~projected/kube-api-access-lmfq7 DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84346da8-55f2-4344-b7d0-dbfb2996f2e7/volumes/kubernetes.io~projected/kube-api-access-szdrr DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7951402aeba57244ffbe2add8b3ef385c13e12c15f4417740d8c6a7b3e06f75d/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5264bed43ef0b66623dec4796da7b3bb76bfbb97bffd3026398a3dccb6a16e40/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/391996f2-6666-4d62-9af2-1291a66828bf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f7ce5e27-0a29-46db-a45d-94dd98cbe059/volumes/kubernetes.io~projected/kube-api-access-cxjtk DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42155cb9-5b97-4aa0-8a18-70af60804e34/volumes/kubernetes.io~projected/kube-api-access-l52zw DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3979a4dbf45de6174e65b75182d05294cd60f1fab756ae5cec10d8f469f33e21/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7212e5ac-eea5-438a-a162-74821cf38b76/volumes/kubernetes.io~projected/kube-api-access-mhkl4 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f941e4e4b67f967090a4170d32b43b293f91cf56a16c0bf5af4ec29cb49c2613/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f29d013-5eaf-42aa-860a-ba18193d7807/volumes/kubernetes.io~projected/kube-api-access-sfpbt DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9/volumes/kubernetes.io~projected/kube-api-access-ftrml DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93ed3e75-7873-4300-be92-0cff9a0d63a7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a56192fcd20a9af9aad204e5d2162d757e993b589a9e3908b4188cb024d17e8/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7e8d3e8247283e32225098487ec34520a2a90cb84181d0b971e199a64baab0e/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/11393f29-8fd7-4a36-b3df-03f0992fc7ad/volumes/kubernetes.io~projected/kube-api-access-wx5hv DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac9c2fad-cc58-4ec5-b169-271b6937af4b/volumes/kubernetes.io~projected/kube-api-access-g8ltg DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/303f050fdbf7b3c43f865d3732ffc24453c1c8fcdc7146738e23217094f5a4f4/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~projected/kube-api-access-fhp65 DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a10d6ef400c2aaada873e76b1aba180329a6c526c265c296884109e30e9dbbe/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f29d013-5eaf-42aa-860a-ba18193d7807/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/02b629c0-be01-461a-88b8-85a69dfad05b/volumes/kubernetes.io~projected/kube-api-access-lmlgr DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/937c4a9525ad300b9baf2e749750324f985f733829dcf4fe937199dd621e013c/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6167377e81c1d4b4f582f023dc700b09a8340d50657f21a7b7a131a41dccd9f3/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8/volumes/kubernetes.io~projected/kube-api-access-d2xjq DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/480b39b91d71a81867488ab2dcb39795bf944e522ca2ad76f0b2934d3e0b9bab/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/800e718b-7419-4df9-836a-cea3686ceda9/volumes/kubernetes.io~projected/kube-api-access-msw47 DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98a66792bd4248f2fdec8187af68dddc5fd7e1057d997408076f5b40c902e33c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50fe92da315b6dc10e5250bca9dcb85546c6e5aa696bf1c3dff174ef9e1bc17e/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c72f864-73b2-48b6-ba6c-fe16323a2d18/volumes/kubernetes.io~projected/kube-api-access-nhgdv DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/93ed3e75-7873-4300-be92-0cff9a0d63a7/volumes/kubernetes.io~projected/kube-api-access-z7k4q DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8db7439dddd8b2f53cd4e6ac413734c8079dfd5e6267ba1170fd427879760fc/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cad7f8eb-8604-4b8b-a921-38b9d5c281db/volumes/kubernetes.io~projected/kube-api-access-prjhg DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cc4e16f243d0dbbf7104b96dc1cfb68747d1ea324b3f55e870ac53b7664d209/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0fd2dfc7fa02952cd90a71420c4870c6f0e1fc33234dec4eac740c3c937504a/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dec71dde8963b69ee2b5f03cd1adb9d18ccc4d80f64dd47757147179b57f692/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1d67bea-af2e-44ad-ab9b-01123c2889fc/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1d67bea-af2e-44ad-ab9b-01123c2889fc/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f1d67bea-af2e-44ad-ab9b-01123c2889fc/volumes/kubernetes.io~projected/kube-api-access-rznb5 DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67fb2af9-df3c-41a8-997c-e16de0871a3e/volumes/kubernetes.io~projected/kube-api-access-wdslj DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5e82585-9fee-4f52-a796-ef82b220e6f6/volumes/kubernetes.io~projected/kube-api-access-95mdj DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/de6988ac-1b54-4229-9424-aa8e28cbb6c6/volumes/kubernetes.io~projected/kube-api-access-nzp62 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67fb2af9-df3c-41a8-997c-e16de0871a3e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef/volumes/kubernetes.io~projected/kube-api-access-wn9vt DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cd01f41-97de-437c-aacf-29cb3668570b/volumes/kubernetes.io~projected/kube-api-access-8rx4n DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a5d5930-3046-4003-bce1-4716ab70ef86/volumes/kubernetes.io~projected/kube-api-access-9z9x5 DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b97e5c9-769c-4c97-8847-50a1d8ad613d/volumes/kubernetes.io~projected/kube-api-access-2ngjf DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c453175eff516c698e66a1b7fca3d1cc5b61ff55636b4bf3f2082ccaa4ff1a9/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5fa96e71655df633bb5f03b03d39f54c418e1841d26a7b50e96f240648c751f5/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e40cc373f4722e46eeee669f2b7f968458827ab0e49b430efa82d3190db4463/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4dbbaf4e2c1247524a00d3c23a0803587d49d72584cd287c53b4e9200982013/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1dec71dde8963b6 MacAddress:8a:9c:e7:03:36:d5 Speed:10000 Mtu:8901} {Name:303f050fdbf7b3c MacAddress:b6:f1:04:af:b6:c7 Speed:10000 Mtu:8901} {Name:3979a4dbf45de61 MacAddress:3a:13:c4:28:48:c1 Speed:10000 Mtu:8901} {Name:480b39b91d71a81 MacAddress:02:70:05:1b:93:4d Speed:10000 Mtu:8901} {Name:4c453175eff516c MacAddress:d6:c9:5e:63:c5:92 Speed:10000 Mtu:8901} {Name:50fe92da315b6dc MacAddress:2a:e4:e2:79:d6:a6 Speed:10000 Mtu:8901} {Name:5fa96e71655df63 MacAddress:12:bf:0a:84:4d:48 Speed:10000 Mtu:8901} {Name:6167377e81c1d4b MacAddress:d2:0a:ac:84:22:d5 Speed:10000 Mtu:8901} {Name:7951402aeba5724 MacAddress:06:09:db:96:95:5c Speed:10000 Mtu:8901} {Name:7e40cc373f4722e MacAddress:2e:47:aa:af:d4:09 Speed:10000 Mtu:8901} {Name:937c4a9525ad300 MacAddress:0e:a7:2c:8b:28:7f Speed:10000 Mtu:8901} {Name:9ab94d7e66d5693 MacAddress:e6:a8:cd:6f:d7:ff Speed:10000 Mtu:8901} {Name:a7e8d3e8247283e MacAddress:96:a3:54:5d:a6:c1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d1:8e:27:63 Speed:0 Mtu:9001} {Name:br-int MacAddress:86:f3:e2:01:8e:b5 Speed:0 Mtu:8901} {Name:d8db7439dddd8b2 MacAddress:22:76:39:a1:3d:83 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d1:8e:27:63 Speed:0 Mtu:9001} {Name:fa33d9537fe0d41 MacAddress:8e:59:23:b4:f7:2f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:5e:3a:59:37:f8:f4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:5a:bd:a8:25:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:53:06.065087 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.064747 5001 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:53:06.065087 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.064824 5001 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:53:06.065335 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065123 5001 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:53:06.065335 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065139 5001 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:53:06.065335 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065262 5001 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:53:06.065335 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065270 5001 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:53:06.065335 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065280 5001 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:53:06.065335 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065294 5001 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:53:06.065335 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065335 5001 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:53:06.065536 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065494 5001 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:53:06.065607 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065597 5001 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:53:06.065645 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065637 5001 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:53:06.065676 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065649 5001 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:53:06.065676 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065658 5001 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:53:06.065676 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065667 5001 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:53:06.065676 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.065676 5001 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:53:06.067724 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.067214 5001 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:53:06.067724 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.067237 5001 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:53:06.067883 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.067839 5001 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:53:06.070368 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070353 5001 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:53:06.070368 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070366 5001 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:53:06.070479 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070452 5001 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 05:53:06.070637 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070625 5001 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:53:06.070668 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070655 5001 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:53:06.070748 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070739 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:53:06.070788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070753 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:53:06.070788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070760 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:53:06.070788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070765 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:53:06.070788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070771 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:53:06.070788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070776 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:53:06.070788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070782 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:53:06.070788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070787 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:53:06.071068 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070794 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:53:06.071068 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070804 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:53:06.071068 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070817 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:53:06.071068 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070827 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:53:06.071068 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070849 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:53:06.071068 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.070856 5001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:53:06.074200 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.074186 5001 server.go:1263] "Started kubelet" Oct 30 05:53:06.074319 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.074260 5001 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:53:06.074377 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.074284 5001 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:53:06.074426 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.074378 5001 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:53:06.074686 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.074667 5001 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:53:06.074753 ip-10-0-128-31 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:53:06.075961 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.075603 5001 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:53:06.080208 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.080020 5001 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:53:06.080596 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.080406 5001 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:53:06.081687 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.080663 5001 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:47:27 +0000 UTC" deadline="2028-05-12 15:55:20.428333189 +0000 UTC" Oct 30 05:53:06.081687 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.080687 5001 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13450h2m14.347649162s" Oct 30 05:53:06.081687 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.080779 5001 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:53:06.081687 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.080797 5001 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:53:06.081687 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.080906 5001 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:53:06.081977 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.081704 5001 factory.go:55] Registering systemd factory Oct 30 05:53:06.081977 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.081725 5001 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:53:06.082217 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:06.082165 5001 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found" Oct 30 05:53:06.084334 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.083778 5001 factory.go:153] Registering CRI-O factory Oct 30 05:53:06.084334 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.083800 5001 factory.go:223] Registration of the crio container factory successfully Oct 30 05:53:06.084334 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.083867 5001 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:53:06.084334 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.083890 5001 factory.go:103] Registering Raw factory Oct 30 05:53:06.084334 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.083906 5001 manager.go:1199] Started watching for new ooms in manager Oct 30 05:53:06.084334 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.084262 5001 manager.go:319] Starting recovery of all containers Oct 30 05:53:06.086018 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.085979 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00361e44-6e15-460e-88e7-1fd0398406e5" volumeName="kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-registry-certificates" seLinuxMountContext="" Oct 30 05:53:06.086097 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086018 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00361e44-6e15-460e-88e7-1fd0398406e5" volumeName="kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-installation-pull-secrets" seLinuxMountContext="" Oct 30 05:53:06.086097 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086032 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cd01f41-97de-437c-aacf-29cb3668570b" volumeName="kubernetes.io/projected/8cd01f41-97de-437c-aacf-29cb3668570b-kube-api-access-8rx4n" seLinuxMountContext="" Oct 30 05:53:06.086097 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086045 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f29d013-5eaf-42aa-860a-ba18193d7807" volumeName="kubernetes.io/projected/8f29d013-5eaf-42aa-860a-ba18193d7807-kube-api-access-sfpbt" seLinuxMountContext="" Oct 30 05:53:06.086097 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086056 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f29d013-5eaf-42aa-860a-ba18193d7807" volumeName="kubernetes.io/secret/8f29d013-5eaf-42aa-860a-ba18193d7807-serving-cert" seLinuxMountContext="" Oct 30 05:53:06.086097 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086066 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1d67bea-af2e-44ad-ab9b-01123c2889fc" volumeName="kubernetes.io/projected/f1d67bea-af2e-44ad-ab9b-01123c2889fc-kube-api-access-rznb5" seLinuxMountContext="" Oct 30 05:53:06.086097 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086077 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ce5e27-0a29-46db-a45d-94dd98cbe059" volumeName="kubernetes.io/configmap/f7ce5e27-0a29-46db-a45d-94dd98cbe059-serviceca" seLinuxMountContext="" Oct 30 05:53:06.086097 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086088 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" volumeName="kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086101 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391996f2-6666-4d62-9af2-1291a66828bf" volumeName="kubernetes.io/configmap/391996f2-6666-4d62-9af2-1291a66828bf-config" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086113 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef" volumeName="kubernetes.io/configmap/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-config" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086124 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1d67bea-af2e-44ad-ab9b-01123c2889fc" volumeName="kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-config" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086136 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184d6270-19ed-4e6b-80c6-9734584b82a1" volumeName="kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-bound-sa-token" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086148 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b97e5c9-769c-4c97-8847-50a1d8ad613d" volumeName="kubernetes.io/projected/1b97e5c9-769c-4c97-8847-50a1d8ad613d-kube-api-access-2ngjf" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086161 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7212e5ac-eea5-438a-a162-74821cf38b76" volumeName="kubernetes.io/empty-dir/7212e5ac-eea5-438a-a162-74821cf38b76-tmp-dir" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086175 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a5d5930-3046-4003-bce1-4716ab70ef86" volumeName="kubernetes.io/configmap/8a5d5930-3046-4003-bce1-4716ab70ef86-iptables-alerter-script" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086186 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a5d5930-3046-4003-bce1-4716ab70ef86" volumeName="kubernetes.io/projected/8a5d5930-3046-4003-bce1-4716ab70ef86-kube-api-access-9z9x5" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086197 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f29d013-5eaf-42aa-860a-ba18193d7807" volumeName="kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-snapshots" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086225 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9c2d4d-67e5-4c4d-8294-86ae9050232a" volumeName="kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-config" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086236 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9c2d4d-67e5-4c4d-8294-86ae9050232a" volumeName="kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-trusted-ca" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086247 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00361e44-6e15-460e-88e7-1fd0398406e5" volumeName="kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-trusted-ca" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086259 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00361e44-6e15-460e-88e7-1fd0398406e5" volumeName="kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-kube-api-access-fhp65" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086272 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42155cb9-5b97-4aa0-8a18-70af60804e34" volumeName="kubernetes.io/projected/42155cb9-5b97-4aa0-8a18-70af60804e34-kube-api-access-l52zw" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086283 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8" volumeName="kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086295 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac9c2fad-cc58-4ec5-b169-271b6937af4b" volumeName="kubernetes.io/projected/ac9c2fad-cc58-4ec5-b169-271b6937af4b-kube-api-access-g8ltg" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086323 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad7f8eb-8604-4b8b-a921-38b9d5c281db" volumeName="kubernetes.io/projected/cad7f8eb-8604-4b8b-a921-38b9d5c281db-kube-api-access-prjhg" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086335 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9c2d4d-67e5-4c4d-8294-86ae9050232a" volumeName="kubernetes.io/secret/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-serving-cert" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086347 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1d67bea-af2e-44ad-ab9b-01123c2889fc" volumeName="kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-env-overrides" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086360 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184d6270-19ed-4e6b-80c6-9734584b82a1" volumeName="kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-certificates" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086376 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="800e718b-7419-4df9-836a-cea3686ceda9" volumeName="kubernetes.io/empty-dir/800e718b-7419-4df9-836a-cea3686ceda9-tmp-dir" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086387 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84346da8-55f2-4344-b7d0-dbfb2996f2e7" volumeName="kubernetes.io/projected/84346da8-55f2-4344-b7d0-dbfb2996f2e7-kube-api-access-szdrr" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086398 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84346da8-55f2-4344-b7d0-dbfb2996f2e7" volumeName="kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-stats-auth" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086409 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a792a513-44ef-43d1-a48c-e17d8f939042" volumeName="kubernetes.io/empty-dir/a792a513-44ef-43d1-a48c-e17d8f939042-data-volume" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086435 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184d6270-19ed-4e6b-80c6-9734584b82a1" volumeName="kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-installation-pull-secrets" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086452 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b6113ea-d183-4a72-a574-80c03c8f5352" volumeName="kubernetes.io/configmap/3b6113ea-d183-4a72-a574-80c03c8f5352-konnectivity-ca" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086465 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42155cb9-5b97-4aa0-8a18-70af60804e34" volumeName="kubernetes.io/configmap/42155cb9-5b97-4aa0-8a18-70af60804e34-telemetry-config" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086476 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67fb2af9-df3c-41a8-997c-e16de0871a3e" volumeName="kubernetes.io/secret/67fb2af9-df3c-41a8-997c-e16de0871a3e-serving-cert" seLinuxMountContext="" Oct 30 05:53:06.086469 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086487 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f29d013-5eaf-42aa-860a-ba18193d7807" volumeName="kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086498 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7e5b861-eb47-4d8a-8974-90a2368489ed" volumeName="kubernetes.io/projected/c7e5b861-eb47-4d8a-8974-90a2368489ed-kube-api-access-p6tct" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086510 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1d67bea-af2e-44ad-ab9b-01123c2889fc" volumeName="kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-script-lib" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086524 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00361e44-6e15-460e-88e7-1fd0398406e5" volumeName="kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-image-registry-private-configuration" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086537 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11393f29-8fd7-4a36-b3df-03f0992fc7ad" volumeName="kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cni-binary-copy" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086550 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef" volumeName="kubernetes.io/projected/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-kube-api-access-wn9vt" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086563 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5e82585-9fee-4f52-a796-ef82b220e6f6" volumeName="kubernetes.io/projected/b5e82585-9fee-4f52-a796-ef82b220e6f6-kube-api-access-95mdj" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086576 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6988ac-1b54-4229-9424-aa8e28cbb6c6" volumeName="kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086588 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1d67bea-af2e-44ad-ab9b-01123c2889fc" volumeName="kubernetes.io/secret/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086603 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391996f2-6666-4d62-9af2-1291a66828bf" volumeName="kubernetes.io/secret/391996f2-6666-4d62-9af2-1291a66828bf-serving-cert" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086615 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" volumeName="kubernetes.io/projected/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-kube-api-access-hgl2n" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086626 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="800e718b-7419-4df9-836a-cea3686ceda9" volumeName="kubernetes.io/configmap/800e718b-7419-4df9-836a-cea3686ceda9-config-volume" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086637 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6988ac-1b54-4229-9424-aa8e28cbb6c6" volumeName="kubernetes.io/projected/de6988ac-1b54-4229-9424-aa8e28cbb6c6-kube-api-access-nzp62" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086649 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d" volumeName="kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-tmp" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086661 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184d6270-19ed-4e6b-80c6-9734584b82a1" volumeName="kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-trusted-ca" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086675 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184d6270-19ed-4e6b-80c6-9734584b82a1" volumeName="kubernetes.io/empty-dir/184d6270-19ed-4e6b-80c6-9734584b82a1-ca-trust-extracted" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086688 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184d6270-19ed-4e6b-80c6-9734584b82a1" volumeName="kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-kube-api-access-krbp7" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086701 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" volumeName="kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-service-ca-bundle" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086714 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67fb2af9-df3c-41a8-997c-e16de0871a3e" volumeName="kubernetes.io/projected/67fb2af9-df3c-41a8-997c-e16de0871a3e-kube-api-access-wdslj" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086729 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6988ac-1b54-4229-9424-aa8e28cbb6c6" volumeName="kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-binary-copy" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086753 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ce5e27-0a29-46db-a45d-94dd98cbe059" volumeName="kubernetes.io/projected/f7ce5e27-0a29-46db-a45d-94dd98cbe059-kube-api-access-cxjtk" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086766 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391996f2-6666-4d62-9af2-1291a66828bf" volumeName="kubernetes.io/projected/391996f2-6666-4d62-9af2-1291a66828bf-kube-api-access-8srpr" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086795 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="800e718b-7419-4df9-836a-cea3686ceda9" volumeName="kubernetes.io/projected/800e718b-7419-4df9-836a-cea3686ceda9-kube-api-access-msw47" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086806 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9" volumeName="kubernetes.io/projected/f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9-kube-api-access-ftrml" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086820 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11393f29-8fd7-4a36-b3df-03f0992fc7ad" volumeName="kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-daemon-config" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086833 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f29d013-5eaf-42aa-860a-ba18193d7807" volumeName="kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-service-ca-bundle" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086846 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93ed3e75-7873-4300-be92-0cff9a0d63a7" volumeName="kubernetes.io/projected/93ed3e75-7873-4300-be92-0cff9a0d63a7-kube-api-access-z7k4q" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086860 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93ed3e75-7873-4300-be92-0cff9a0d63a7" volumeName="kubernetes.io/secret/93ed3e75-7873-4300-be92-0cff9a0d63a7-cert" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086873 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9c2d4d-67e5-4c4d-8294-86ae9050232a" volumeName="kubernetes.io/projected/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-kube-api-access-vzpmr" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086886 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11393f29-8fd7-4a36-b3df-03f0992fc7ad" volumeName="kubernetes.io/projected/11393f29-8fd7-4a36-b3df-03f0992fc7ad-kube-api-access-wx5hv" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086898 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c72f864-73b2-48b6-ba6c-fe16323a2d18" volumeName="kubernetes.io/projected/6c72f864-73b2-48b6-ba6c-fe16323a2d18-kube-api-access-nhgdv" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086910 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6d3694c-d99c-43d1-be3f-6df16717e1de" volumeName="kubernetes.io/configmap/d6d3694c-d99c-43d1-be3f-6df16717e1de-nginx-conf" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086921 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02b629c0-be01-461a-88b8-85a69dfad05b" volumeName="kubernetes.io/projected/02b629c0-be01-461a-88b8-85a69dfad05b-kube-api-access-lmlgr" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086937 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b6113ea-d183-4a72-a574-80c03c8f5352" volumeName="kubernetes.io/secret/3b6113ea-d183-4a72-a574-80c03c8f5352-agent-certs" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086949 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" volumeName="kubernetes.io/empty-dir/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-archives-path" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086964 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84346da8-55f2-4344-b7d0-dbfb2996f2e7" volumeName="kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-default-certificate" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086975 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f29d013-5eaf-42aa-860a-ba18193d7807" volumeName="kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-tmp" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086986 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00361e44-6e15-460e-88e7-1fd0398406e5" volumeName="kubernetes.io/empty-dir/00361e44-6e15-460e-88e7-1fd0398406e5-ca-trust-extracted" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.086998 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00361e44-6e15-460e-88e7-1fd0398406e5" volumeName="kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-bound-sa-token" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.087010 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184d6270-19ed-4e6b-80c6-9734584b82a1" volumeName="kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-image-registry-private-configuration" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.087022 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67fb2af9-df3c-41a8-997c-e16de0871a3e" volumeName="kubernetes.io/configmap/67fb2af9-df3c-41a8-997c-e16de0871a3e-config" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.087034 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7212e5ac-eea5-438a-a162-74821cf38b76" volumeName="kubernetes.io/projected/7212e5ac-eea5-438a-a162-74821cf38b76-kube-api-access-mhkl4" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.087045 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a792a513-44ef-43d1-a48c-e17d8f939042" volumeName="kubernetes.io/projected/a792a513-44ef-43d1-a48c-e17d8f939042-kube-api-access-lmfq7" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.087056 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6988ac-1b54-4229-9424-aa8e28cbb6c6" volumeName="kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.087067 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d" volumeName="kubernetes.io/projected/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-kube-api-access-28fpx" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.087078 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d44df296-4186-4c40-81de-1ed06c15955e" volumeName="kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.087090 5001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d" volumeName="kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-tuned" seLinuxMountContext="" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.087102 5001 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:53:06.087875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.087110 5001 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:53:06.100625 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.100607 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 05:53:06.100830 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.100813 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 05:53:06.107504 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.107473 5001 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:53:06.108869 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.108847 5001 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:53:06.108973 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.108877 5001 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:53:06.108973 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.108901 5001 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:53:06.108973 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:06.108943 5001 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 05:53:06.124701 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.124665 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 05:53:06.124815 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.124699 5001 generic.go:356] "Generic (PLEG): container finished" podID="cb9c2d4d-67e5-4c4d-8294-86ae9050232a" containerID="a1b0e58f3452ccffe96b15479c7192f31726ad35f4c7c369ea791c73d61053fe" exitCode=255 Oct 30 05:53:06.126611 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.126591 5001 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 05:53:06.127546 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.127530 5001 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:06.127792 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.127775 5001 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:06.128395 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.128371 5001 generic.go:356] "Generic (PLEG): container finished" podID="1bb6269644393a710c6775a446a340ab" containerID="62c14a981bdb43f4e95d9f612272ce4648e5c816224421cb627efe5de9874865" exitCode=0 Oct 30 05:53:06.139721 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.139698 5001 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:06.140082 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.140064 5001 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:06.162139 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.162076 5001 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="6d13c54a0a19a721dad4c7218b2e8dec4dc6896edca9fc8de7abe899762dbf03" exitCode=0 Oct 30 05:53:06.162139 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.162099 5001 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="9d308222186dd9a3ba6359b4412dab6472f7841c46ebd64fa4a590ed897079ea" exitCode=0 Oct 30 05:53:06.162139 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.162107 5001 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="eda7ff193b1121c44447cdc508f8cc42be5dba763a2abb45e1160457608d3734" exitCode=0 Oct 30 05:53:06.162139 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.162112 5001 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="bfaf8bf4d86fbe25bfb02dfab0fce1c82d299176a46fc72b048fff24f4fe17a0" exitCode=0 Oct 30 05:53:06.162139 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.162118 5001 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="ecbd10b7176d719993bc30656c927c4174abd95b3977df172dd25784ac3f94ec" exitCode=0 Oct 30 05:53:06.162139 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.162123 5001 generic.go:356] "Generic (PLEG): container finished" podID="de6988ac-1b54-4229-9424-aa8e28cbb6c6" containerID="cccafdb09c457277de081157c9eeb3cf7c955083151fda3fa533313f24236185" exitCode=0 Oct 30 05:53:06.186229 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.186211 5001 manager.go:324] Recovery completed Oct 30 05:53:06.207093 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207070 5001 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:53:06.207093 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207085 5001 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:53:06.207240 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207100 5001 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:53:06.207274 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207239 5001 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 05:53:06.207274 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207250 5001 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 05:53:06.207274 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207265 5001 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 05:53:06.207274 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207270 5001 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 05:53:06.207274 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207276 5001 policy_none.go:50] "Start" Oct 30 05:53:06.207434 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207288 5001 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:53:06.207434 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207296 5001 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:53:06.207485 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207447 5001 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 05:53:06.207485 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207455 5001 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 05:53:06.207485 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.207460 5001 policy_none.go:44] "Start" Oct 30 05:53:06.209239 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:06.209220 5001 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 05:53:06.211700 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.211676 5001 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:53:06.214109 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.214093 5001 manager.go:341] "Starting Device Plugin manager" Oct 30 05:53:06.214590 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:06.214123 5001 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:53:06.214671 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.214592 5001 server.go:87] "Starting device plugin registration server" Oct 30 05:53:06.214882 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.214868 5001 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:53:06.214946 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.214884 5001 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:53:06.215007 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.214962 5001 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:53:06.215159 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.215099 5001 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:53:06.215241 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.215170 5001 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:53:06.215241 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.215178 5001 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:53:06.215343 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.215240 5001 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:53:06.21515779Z","UUID":"3b087541-2102-4475-9215-4798dfba341f","Handler":null,"Name":"","Endpoint":""} Oct 30 05:53:06.216610 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:06.216589 5001 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:53:06.216748 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.216730 5001 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:53:06.216846 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.216757 5001 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:53:06.315574 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.315541 5001 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:53:06.317118 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.317102 5001 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:53:06.317193 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.317129 5001 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:53:06.317193 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.317138 5001 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:53:06.317193 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.317176 5001 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.350726 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.350702 5001 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.350822 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.350759 5001 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.409448 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.409409 5001 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"] Oct 30 05:53:06.409996 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.409967 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerStarted","Data":"9844588068de4d8f2b7e4add7e0eb38e65d73c7fac039792c734686970ea2298"} Oct 30 05:53:06.410086 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.410003 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerDied","Data":"62c14a981bdb43f4e95d9f612272ce4648e5c816224421cb627efe5de9874865"} Oct 30 05:53:06.410086 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.410027 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerStarted","Data":"f4dbbaf4e2c1247524a00d3c23a0803587d49d72584cd287c53b4e9200982013"} Oct 30 05:53:06.410086 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.410046 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" event={"ID":"ed322e80a89b2bbeafbf0f88fe9597d6","Type":"ContainerStarted","Data":"9f8bb74f3dff7f361bf929c7fbe58244c6deac571caa8e58d17ced38e59f8533"} Oct 30 05:53:06.410086 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.410059 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" event={"ID":"ed322e80a89b2bbeafbf0f88fe9597d6","Type":"ContainerStarted","Data":"5264bed43ef0b66623dec4796da7b3bb76bfbb97bffd3026398a3dccb6a16e40"} Oct 30 05:53:06.410273 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.410127 5001 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.410273 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.410143 5001 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.437561 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.437517 5001 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:06.437561 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:06.437554 5001 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.439743 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.439729 5001 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:06.439790 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:06.439764 5001 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.529100 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.529071 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.529100 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.529104 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.529267 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.529121 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.630276 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.630239 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.630276 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.630261 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.630477 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.630289 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.630477 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.630354 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.630477 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.630393 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:53:06.630477 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:06.630402 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:53:07.066396 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.066326 5001 apiserver.go:52] "Watching apiserver" Oct 30 05:53:07.110397 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.110363 5001 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:53:07.111098 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.111066 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-r5rsl","kube-system/global-pull-secret-syncer-bnv8q","openshift-image-registry/image-registry-7b9c9cf4b4-89zvd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz","openshift-multus/multus-additional-cni-plugins-chl67","openshift-ingress/router-default-5b994dfc85-2v9ww","kube-system/kas-connection-checker-685895c7c9-j7pwr","openshift-ingress-canary/ingress-canary-lrww4","kube-system/konnectivity-agent-cq5xz","openshift-image-registry/node-ca-l652g","openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft","openshift-insights/insights-runtime-extractor-dtvjb","openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z","openshift-console-operator/console-operator-7cd56b8dbc-7ztmq","openshift-insights/periodic-gathering-ghpdk-jn8g9","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr","kube-system/kas-connection-checker-685895c7c9-74dvn","openshift-network-console/networking-console-plugin-86c478bb88-5twq8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal","openshift-cluster-node-tuning-operator/tuned-85kft","openshift-network-operator/iptables-alerter-wmrrw","openshift-ovn-kubernetes/ovnkube-node-zzk85","openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf","openshift-network-diagnostics/network-check-target-xsrdq","kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal","openshift-dns/dns-default-5gv9v","kube-system/kas-connection-checker-685895c7c9-2626d","openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq","openshift-dns/node-resolver-dz874","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w","openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr","openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj","openshift-multus/multus-86c7b","openshift-insights/insights-operator-7c4986d449-lq9h9"] Oct 30 05:53:07.112674 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.112650 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.113633 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.113614 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:53:07.113946 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.113925 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:53:07.115499 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.115263 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.116584 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.116560 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:07.118752 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.118727 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:07.123380 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.123364 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.124292 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.124277 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:07.126739 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.126721 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.137268 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.137251 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 05:53:07.137399 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.137375 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.137436 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.137430 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.138684 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.138669 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.138784 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.138772 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 05:53:07.138992 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.138979 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.139033 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.139021 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.142773 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.142760 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 05:53:07.147425 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.147404 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 05:53:07.147554 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.147494 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 05:53:07.147554 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.147503 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 05:53:07.147676 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.147658 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 05:53:07.149282 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149265 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:53:07.149282 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149272 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 05:53:07.149432 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149289 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d9dv5\"" Oct 30 05:53:07.149432 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149298 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 05:53:07.149534 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149472 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 05:53:07.149534 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149522 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:53:07.149614 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149533 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:53:07.149614 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149546 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vzk7w\"" Oct 30 05:53:07.149614 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149547 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6k8z9\"" Oct 30 05:53:07.149732 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149664 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q2ncw\"" Oct 30 05:53:07.149732 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149690 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 05:53:07.149732 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149709 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 05:53:07.149732 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.149701 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-5fdqr\"" Oct 30 05:53:07.152833 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.152817 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:53:07.154167 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154150 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:53:07.154237 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154173 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l4lh5\"" Oct 30 05:53:07.154237 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154182 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mkzdc\"" Oct 30 05:53:07.154237 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154211 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:53:07.154409 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154241 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 05:53:07.154409 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154266 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 05:53:07.154409 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154266 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:53:07.154409 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154379 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mpzcb\"" Oct 30 05:53:07.154409 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154403 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dk4qd\"" Oct 30 05:53:07.154656 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154462 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5hkvn\"" Oct 30 05:53:07.154656 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154489 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xln8v\"" Oct 30 05:53:07.154656 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154492 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r6cnz\"" Oct 30 05:53:07.154656 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154529 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 05:53:07.154656 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154558 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-plsgv\"" Oct 30 05:53:07.154858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154746 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q46lp\"" Oct 30 05:53:07.154858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154789 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntdkp\"" Oct 30 05:53:07.154858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154854 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x8h66\"" Oct 30 05:53:07.155003 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154853 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.155003 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154895 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wkwzp\"" Oct 30 05:53:07.155003 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154896 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-99gsm\"" Oct 30 05:53:07.155003 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154939 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 05:53:07.155003 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.154985 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-27cwd\"" Oct 30 05:53:07.155915 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.155899 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.156043 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.156031 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.156247 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.156231 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:53:07.156336 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.156248 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.156650 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.156634 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.156729 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.156657 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.156810 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.156791 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.156961 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.156944 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.157239 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.157224 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:53:07.157390 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.157372 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.157729 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.157706 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:53:07.157993 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.157968 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.158076 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.158053 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:53:07.158470 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.158453 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.158752 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.158734 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.158918 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.158893 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.158918 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.158909 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:53:07.159018 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.158899 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.159018 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159009 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.159091 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.158898 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 05:53:07.159157 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159142 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.159225 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159175 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.159323 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159287 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.159488 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159471 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 05:53:07.159564 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159536 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.159634 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159619 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:53:07.159719 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159702 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.159768 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159730 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 05:53:07.159768 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159759 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.159899 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159886 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.159948 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.159908 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.160084 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160057 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.160084 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160077 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:53:07.160204 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160124 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.160333 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160299 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.160407 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160334 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.160407 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160346 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.160511 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160441 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:53:07.160511 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160466 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:53:07.160511 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160507 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.160634 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160536 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:53:07.160634 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160537 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.160634 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160577 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.160760 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160687 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:53:07.160760 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160720 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 05:53:07.160760 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160750 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 05:53:07.160879 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160768 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 05:53:07.160879 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.160865 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:53:07.161100 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.161086 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:53:07.161255 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.161106 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 05:53:07.161346 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.161328 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:53:07.165932 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.165919 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 05:53:07.166098 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.166084 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 05:53:07.174109 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.174093 5001 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:53:07.174229 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.174217 5001 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:53:07.181034 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.181013 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-7qmpr_f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9/migrator/0.log" Oct 30 05:53:07.182761 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.182746 5001 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:53:07.198188 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.198167 5001 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:07.198263 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.198203 5001 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" Oct 30 05:53:07.202691 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.202677 5001 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:07.202739 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.202712 5001 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" Oct 30 05:53:07.234786 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.234766 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prjhg\" (UniqueName: \"kubernetes.io/projected/cad7f8eb-8604-4b8b-a921-38b9d5c281db-kube-api-access-prjhg\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:07.234886 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.234792 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srpr\" (UniqueName: \"kubernetes.io/projected/391996f2-6666-4d62-9af2-1291a66828bf-kube-api-access-8srpr\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:53:07.234886 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.234810 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgl2n\" (UniqueName: \"kubernetes.io/projected/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-kube-api-access-hgl2n\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:07.234886 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.234826 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7ce5e27-0a29-46db-a45d-94dd98cbe059-serviceca\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:53:07.234886 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.234842 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhp65\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-kube-api-access-fhp65\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.234886 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.234860 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-conf\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.234886 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.234880 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:53:07.235202 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.234903 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-stats-auth\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.235202 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.234930 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-conf-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.235202 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.234958 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93ed3e75-7873-4300-be92-0cff9a0d63a7-cert\") pod \"ingress-canary-lrww4\" (UID: \"93ed3e75-7873-4300-be92-0cff9a0d63a7\") " pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:53:07.235202 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.234981 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-slash\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.235202 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235002 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.235202 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235041 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-trusted-ca\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.235202 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235073 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:07.235202 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235106 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.235202 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235172 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7ce5e27-0a29-46db-a45d-94dd98cbe059-serviceca\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:53:07.235202 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235185 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42155cb9-5b97-4aa0-8a18-70af60804e34-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235246 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a5d5930-3046-4003-bce1-4716ab70ef86-iptables-alerter-script\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235253 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-stats-auth\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235284 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93ed3e75-7873-4300-be92-0cff9a0d63a7-cert\") pod \"ingress-canary-lrww4\" (UID: \"93ed3e75-7873-4300-be92-0cff9a0d63a7\") " pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235293 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-tmp\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235257 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d44df296-4186-4c40-81de-1ed06c15955e-original-pull-secret\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235434 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cnibin\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235472 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-tmp\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235493 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-socket-dir-parent\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235530 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-var-lib-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235542 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42155cb9-5b97-4aa0-8a18-70af60804e34-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235547 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a5d5930-3046-4003-bce1-4716ab70ef86-iptables-alerter-script\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235581 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ce5e27-0a29-46db-a45d-94dd98cbe059-host\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235614 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67fb2af9-df3c-41a8-997c-e16de0871a3e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:53:07.235636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235635 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235640 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-trusted-ca\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235653 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-kubelet-config\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235668 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235703 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235713 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67fb2af9-df3c-41a8-997c-e16de0871a3e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235749 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235785 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-bound-sa-token\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235844 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235893 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800e718b-7419-4df9-836a-cea3686ceda9-tmp-dir\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235913 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235929 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-bound-sa-token\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235945 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-image-registry-private-configuration\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235971 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/184d6270-19ed-4e6b-80c6-9734584b82a1-ca-trust-extracted\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.235975 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236015 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-k8s-cni-cncf-io\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236042 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800e718b-7419-4df9-836a-cea3686ceda9-tmp-dir\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236041 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/391996f2-6666-4d62-9af2-1291a66828bf-serving-cert\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236089 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/184d6270-19ed-4e6b-80c6-9734584b82a1-ca-trust-extracted\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236098 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-node-log\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236148 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7212e5ac-eea5-438a-a162-74821cf38b76-hosts-file\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236155 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-image-registry-private-configuration\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236165 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236173 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cnibin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236215 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b6113ea-d183-4a72-a574-80c03c8f5352-agent-certs\") pod \"konnectivity-agent-cq5xz\" (UID: \"3b6113ea-d183-4a72-a574-80c03c8f5352\") " pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236241 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/391996f2-6666-4d62-9af2-1291a66828bf-serving-cert\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:53:07.236329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236278 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-trusted-ca-bundle\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236365 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a5d5930-3046-4003-bce1-4716ab70ef86-host-slash\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236410 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpmr\" (UniqueName: \"kubernetes.io/projected/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-kube-api-access-vzpmr\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236438 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a792a513-44ef-43d1-a48c-e17d8f939042-data-volume\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236440 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b6113ea-d183-4a72-a574-80c03c8f5352-agent-certs\") pod \"konnectivity-agent-cq5xz\" (UID: \"3b6113ea-d183-4a72-a574-80c03c8f5352\") " pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236463 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7212e5ac-eea5-438a-a162-74821cf38b76-tmp-dir\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236494 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a792a513-44ef-43d1-a48c-e17d8f939042-data-volume\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236510 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfq7\" (UniqueName: \"kubernetes.io/projected/a792a513-44ef-43d1-a48c-e17d8f939042-kube-api-access-lmfq7\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236524 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7212e5ac-eea5-438a-a162-74821cf38b76-tmp-dir\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236533 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-image-registry-private-configuration\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236551 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-installation-pull-secrets\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236566 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-snapshots\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236583 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-system-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236605 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236661 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236693 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52zw\" (UniqueName: \"kubernetes.io/projected/42155cb9-5b97-4aa0-8a18-70af60804e34-kube-api-access-l52zw\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236718 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xjq\" (UniqueName: \"kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq\") pod \"network-check-target-xsrdq\" (UID: \"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8\") " pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236737 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-image-registry-private-configuration\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236742 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-registry-certificates\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236746 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-installation-pull-secrets\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236672 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-snapshots\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236775 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-tuned\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236803 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236835 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-tuned\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236840 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236855 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxjtk\" (UniqueName: \"kubernetes.io/projected/f7ce5e27-0a29-46db-a45d-94dd98cbe059-kube-api-access-cxjtk\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236877 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-registry-certificates\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236896 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-dbus\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.236929 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-trusted-ca\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237038 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szdrr\" (UniqueName: \"kubernetes.io/projected/84346da8-55f2-4344-b7d0-dbfb2996f2e7-kube-api-access-szdrr\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237069 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z9x5\" (UniqueName: \"kubernetes.io/projected/8a5d5930-3046-4003-bce1-4716ab70ef86-kube-api-access-9z9x5\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237098 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237098 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-trusted-ca-bundle\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237128 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-tmp\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237165 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-socket-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237170 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-trusted-ca\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237186 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-registration-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237185 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f29d013-5eaf-42aa-860a-ba18193d7807-tmp\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237237 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00361e44-6e15-460e-88e7-1fd0398406e5-ca-trust-extracted\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237264 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6d3694c-d99c-43d1-be3f-6df16717e1de-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237292 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00361e44-6e15-460e-88e7-1fd0398406e5-ca-trust-extracted\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237290 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7k4q\" (UniqueName: \"kubernetes.io/projected/93ed3e75-7873-4300-be92-0cff9a0d63a7-kube-api-access-z7k4q\") pod \"ingress-canary-lrww4\" (UID: \"93ed3e75-7873-4300-be92-0cff9a0d63a7\") " pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237345 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-script-lib\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237397 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-certificates\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237424 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msw47\" (UniqueName: \"kubernetes.io/projected/800e718b-7419-4df9-836a-cea3686ceda9-kube-api-access-msw47\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237448 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzp62\" (UniqueName: \"kubernetes.io/projected/de6988ac-1b54-4229-9424-aa8e28cbb6c6-kube-api-access-nzp62\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237491 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6d3694c-d99c-43d1-be3f-6df16717e1de-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237492 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rznb5\" (UniqueName: \"kubernetes.io/projected/f1d67bea-af2e-44ad-ab9b-01123c2889fc-kube-api-access-rznb5\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237536 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237562 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237605 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbp7\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-kube-api-access-krbp7\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237643 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-script-lib\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237661 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-certificates\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237694 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f29d013-5eaf-42aa-860a-ba18193d7807-serving-cert\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237713 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-multus-certs\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237728 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-sys-fs\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237763 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-kubelet\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237782 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysconfig\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237784 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f29d013-5eaf-42aa-860a-ba18193d7807-serving-cert\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237796 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-bin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237829 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-hostroot\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237852 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlgr\" (UniqueName: \"kubernetes.io/projected/02b629c0-be01-461a-88b8-85a69dfad05b-kube-api-access-lmlgr\") pod \"kas-connection-checker-685895c7c9-74dvn\" (UID: \"02b629c0-be01-461a-88b8-85a69dfad05b\") " pod="kube-system/kas-connection-checker-685895c7c9-74dvn" Oct 30 05:53:07.237849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237868 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237889 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdslj\" (UniqueName: \"kubernetes.io/projected/67fb2af9-df3c-41a8-997c-e16de0871a3e-kube-api-access-wdslj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237929 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mdj\" (UniqueName: \"kubernetes.io/projected/b5e82585-9fee-4f52-a796-ef82b220e6f6-kube-api-access-95mdj\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237946 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6tct\" (UniqueName: \"kubernetes.io/projected/c7e5b861-eb47-4d8a-8974-90a2368489ed-kube-api-access-p6tct\") pod \"kas-connection-checker-685895c7c9-2626d\" (UID: \"c7e5b861-eb47-4d8a-8974-90a2368489ed\") " pod="kube-system/kas-connection-checker-685895c7c9-2626d" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237968 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.237993 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238028 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238115 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cni-binary-copy\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238234 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rx4n\" (UniqueName: \"kubernetes.io/projected/8cd01f41-97de-437c-aacf-29cb3668570b-kube-api-access-8rx4n\") pod \"kas-connection-checker-685895c7c9-j7pwr\" (UID: \"8cd01f41-97de-437c-aacf-29cb3668570b\") " pod="kube-system/kas-connection-checker-685895c7c9-j7pwr" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238256 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238276 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-etc-kubernetes\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238300 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238343 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrml\" (UniqueName: \"kubernetes.io/projected/f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9-kube-api-access-ftrml\") pod \"migrator-66df5d496b-7qmpr\" (UID: \"f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238351 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cni-binary-copy\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238433 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-netns\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238457 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-systemd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238478 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/de6988ac-1b54-4229-9424-aa8e28cbb6c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238486 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a792a513-44ef-43d1-a48c-e17d8f939042-crio-socket\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238523 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpbt\" (UniqueName: \"kubernetes.io/projected/8f29d013-5eaf-42aa-860a-ba18193d7807-kube-api-access-sfpbt\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238561 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-system-cni-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238578 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-bin\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238594 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhgdv\" (UniqueName: \"kubernetes.io/projected/6c72f864-73b2-48b6-ba6c-fe16323a2d18-kube-api-access-nhgdv\") pod \"migrator-66df5d496b-mjhts\" (UID: \"6c72f864-73b2-48b6-ba6c-fe16323a2d18\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238696 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67fb2af9-df3c-41a8-997c-e16de0871a3e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238751 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-run\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238786 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-config\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238850 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67fb2af9-df3c-41a8-997c-e16de0871a3e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238853 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovn-node-metrics-cert\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238884 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-kubernetes\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238933 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn9vt\" (UniqueName: \"kubernetes.io/projected/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-kube-api-access-wn9vt\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238963 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-modprobe-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238985 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovnkube-config\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.238987 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-host\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239015 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-etc-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239033 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1d67bea-af2e-44ad-ab9b-01123c2889fc-ovn-node-metrics-cert\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239117 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239152 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-netd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239179 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239245 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/800e718b-7419-4df9-836a-cea3686ceda9-config-volume\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239271 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-var-lib-kubelet\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239298 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-config\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239411 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/800e718b-7419-4df9-836a-cea3686ceda9-config-volume\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239426 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-serving-cert\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239459 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-default-certificate\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239468 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-config\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239477 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-device-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239510 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-installation-pull-secrets\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239525 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-systemd\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239541 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-service-ca-bundle\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239557 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-systemd-units\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239574 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239590 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-env-overrides\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239631 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239645 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-default-certificate\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239683 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhkl4\" (UniqueName: \"kubernetes.io/projected/7212e5ac-eea5-438a-a162-74821cf38b76-kube-api-access-mhkl4\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239711 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-lib-modules\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239733 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b6113ea-d183-4a72-a574-80c03c8f5352-konnectivity-ca\") pod \"konnectivity-agent-cq5xz\" (UID: \"3b6113ea-d183-4a72-a574-80c03c8f5352\") " pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239745 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-installation-pull-secrets\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239772 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1d67bea-af2e-44ad-ab9b-01123c2889fc-env-overrides\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239845 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/391996f2-6666-4d62-9af2-1291a66828bf-config\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239879 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-log-socket\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239905 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-archives-path\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239909 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-serving-cert\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239930 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-trusted-ca\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239976 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-kubelet\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.239985 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b6113ea-d183-4a72-a574-80c03c8f5352-konnectivity-ca\") pod \"konnectivity-agent-cq5xz\" (UID: \"3b6113ea-d183-4a72-a574-80c03c8f5352\") " pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240021 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8ltg\" (UniqueName: \"kubernetes.io/projected/ac9c2fad-cc58-4ec5-b169-271b6937af4b-kube-api-access-g8ltg\") pod \"volume-data-source-validator-58c9f5f46d-2ql6w\" (UID: \"ac9c2fad-cc58-4ec5-b169-271b6937af4b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240035 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-archives-path\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240056 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/391996f2-6666-4d62-9af2-1291a66828bf-config\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240070 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-service-ca-bundle\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240069 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f29d013-5eaf-42aa-860a-ba18193d7807-service-ca-bundle\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240099 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ngjf\" (UniqueName: \"kubernetes.io/projected/1b97e5c9-769c-4c97-8847-50a1d8ad613d-kube-api-access-2ngjf\") pod \"network-check-source-6f78d59f99-7jsft\" (UID: \"1b97e5c9-769c-4c97-8847-50a1d8ad613d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft" Oct 30 05:53:07.240095 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240133 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-sys\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.242118 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240177 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-netns\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.242118 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240204 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-multus\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.242118 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240225 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-trusted-ca\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.242118 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240325 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx5hv\" (UniqueName: \"kubernetes.io/projected/11393f29-8fd7-4a36-b3df-03f0992fc7ad-kube-api-access-wx5hv\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.242118 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240368 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-ovn\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.242118 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240399 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28fpx\" (UniqueName: \"kubernetes.io/projected/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-kube-api-access-28fpx\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.242118 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240424 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-daemon-config\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.242118 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240563 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-service-ca-bundle\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:07.242118 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.240688 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-daemon-config\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.280035 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.280007 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srpr\" (UniqueName: \"kubernetes.io/projected/391996f2-6666-4d62-9af2-1291a66828bf-kube-api-access-8srpr\") pod \"service-ca-operator-544d749c8c-6rbgq\" (UID: \"391996f2-6666-4d62-9af2-1291a66828bf\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" Oct 30 05:53:07.280235 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.280218 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szdrr\" (UniqueName: \"kubernetes.io/projected/84346da8-55f2-4344-b7d0-dbfb2996f2e7-kube-api-access-szdrr\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.280296 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.280247 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpmr\" (UniqueName: \"kubernetes.io/projected/cb9c2d4d-67e5-4c4d-8294-86ae9050232a-kube-api-access-vzpmr\") pod \"console-operator-7cd56b8dbc-7ztmq\" (UID: \"cb9c2d4d-67e5-4c4d-8294-86ae9050232a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:07.284548 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.284529 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prjhg\" (UniqueName: \"kubernetes.io/projected/cad7f8eb-8604-4b8b-a921-38b9d5c281db-kube-api-access-prjhg\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:07.284640 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.284602 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfq7\" (UniqueName: \"kubernetes.io/projected/a792a513-44ef-43d1-a48c-e17d8f939042-kube-api-access-lmfq7\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.285487 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.285464 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-bound-sa-token\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.285592 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.285571 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52zw\" (UniqueName: \"kubernetes.io/projected/42155cb9-5b97-4aa0-8a18-70af60804e34-kube-api-access-l52zw\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:53:07.285640 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.285612 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-bound-sa-token\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.288787 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.288769 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z9x5\" (UniqueName: \"kubernetes.io/projected/8a5d5930-3046-4003-bce1-4716ab70ef86-kube-api-access-9z9x5\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:53:07.289504 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.289484 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgl2n\" (UniqueName: \"kubernetes.io/projected/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-kube-api-access-hgl2n\") pod \"periodic-gathering-ghpdk-jn8g9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:07.290240 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.290221 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xjq\" (UniqueName: \"kubernetes.io/projected/4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8-kube-api-access-d2xjq\") pod \"network-check-target-xsrdq\" (UID: \"4fbf9447-7ab2-49e9-ae50-99ec6a57e3c8\") " pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:53:07.290797 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.290781 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhp65\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-kube-api-access-fhp65\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.294627 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.294607 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxjtk\" (UniqueName: \"kubernetes.io/projected/f7ce5e27-0a29-46db-a45d-94dd98cbe059-kube-api-access-cxjtk\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:53:07.299760 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.299739 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlgr\" (UniqueName: \"kubernetes.io/projected/02b629c0-be01-461a-88b8-85a69dfad05b-kube-api-access-lmlgr\") pod \"kas-connection-checker-685895c7c9-74dvn\" (UID: \"02b629c0-be01-461a-88b8-85a69dfad05b\") " pod="kube-system/kas-connection-checker-685895c7c9-74dvn" Oct 30 05:53:07.299971 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.299953 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6tct\" (UniqueName: \"kubernetes.io/projected/c7e5b861-eb47-4d8a-8974-90a2368489ed-kube-api-access-p6tct\") pod \"kas-connection-checker-685895c7c9-2626d\" (UID: \"c7e5b861-eb47-4d8a-8974-90a2368489ed\") " pod="kube-system/kas-connection-checker-685895c7c9-2626d" Oct 30 05:53:07.300898 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.300878 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn9vt\" (UniqueName: \"kubernetes.io/projected/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-kube-api-access-wn9vt\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:53:07.301119 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.301102 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhgdv\" (UniqueName: \"kubernetes.io/projected/6c72f864-73b2-48b6-ba6c-fe16323a2d18-kube-api-access-nhgdv\") pod \"migrator-66df5d496b-mjhts\" (UID: \"6c72f864-73b2-48b6-ba6c-fe16323a2d18\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mjhts" Oct 30 05:53:07.301615 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.301598 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8ltg\" (UniqueName: \"kubernetes.io/projected/ac9c2fad-cc58-4ec5-b169-271b6937af4b-kube-api-access-g8ltg\") pod \"volume-data-source-validator-58c9f5f46d-2ql6w\" (UID: \"ac9c2fad-cc58-4ec5-b169-271b6937af4b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2ql6w" Oct 30 05:53:07.302024 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.302004 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdslj\" (UniqueName: \"kubernetes.io/projected/67fb2af9-df3c-41a8-997c-e16de0871a3e-kube-api-access-wdslj\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7chrf\" (UID: \"67fb2af9-df3c-41a8-997c-e16de0871a3e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" Oct 30 05:53:07.303751 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.303733 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rx4n\" (UniqueName: \"kubernetes.io/projected/8cd01f41-97de-437c-aacf-29cb3668570b-kube-api-access-8rx4n\") pod \"kas-connection-checker-685895c7c9-j7pwr\" (UID: \"8cd01f41-97de-437c-aacf-29cb3668570b\") " pod="kube-system/kas-connection-checker-685895c7c9-j7pwr" Oct 30 05:53:07.305245 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.305226 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rznb5\" (UniqueName: \"kubernetes.io/projected/f1d67bea-af2e-44ad-ab9b-01123c2889fc-kube-api-access-rznb5\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.305572 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.305555 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28fpx\" (UniqueName: \"kubernetes.io/projected/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-kube-api-access-28fpx\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.305684 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.305662 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbp7\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-kube-api-access-krbp7\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.305763 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.305750 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mdj\" (UniqueName: \"kubernetes.io/projected/b5e82585-9fee-4f52-a796-ef82b220e6f6-kube-api-access-95mdj\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.307251 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.307234 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msw47\" (UniqueName: \"kubernetes.io/projected/800e718b-7419-4df9-836a-cea3686ceda9-kube-api-access-msw47\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:07.308905 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.308890 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpbt\" (UniqueName: \"kubernetes.io/projected/8f29d013-5eaf-42aa-860a-ba18193d7807-kube-api-access-sfpbt\") pod \"insights-operator-7c4986d449-lq9h9\" (UID: \"8f29d013-5eaf-42aa-860a-ba18193d7807\") " pod="openshift-insights/insights-operator-7c4986d449-lq9h9" Oct 30 05:53:07.308941 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.308905 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzp62\" (UniqueName: \"kubernetes.io/projected/de6988ac-1b54-4229-9424-aa8e28cbb6c6-kube-api-access-nzp62\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.310789 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.310773 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhkl4\" (UniqueName: \"kubernetes.io/projected/7212e5ac-eea5-438a-a162-74821cf38b76-kube-api-access-mhkl4\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:53:07.310968 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.310948 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ngjf\" (UniqueName: \"kubernetes.io/projected/1b97e5c9-769c-4c97-8847-50a1d8ad613d-kube-api-access-2ngjf\") pod \"network-check-source-6f78d59f99-7jsft\" (UID: \"1b97e5c9-769c-4c97-8847-50a1d8ad613d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7jsft" Oct 30 05:53:07.310999 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.310951 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7k4q\" (UniqueName: \"kubernetes.io/projected/93ed3e75-7873-4300-be92-0cff9a0d63a7-kube-api-access-z7k4q\") pod \"ingress-canary-lrww4\" (UID: \"93ed3e75-7873-4300-be92-0cff9a0d63a7\") " pod="openshift-ingress-canary/ingress-canary-lrww4" Oct 30 05:53:07.312441 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.312425 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx5hv\" (UniqueName: \"kubernetes.io/projected/11393f29-8fd7-4a36-b3df-03f0992fc7ad-kube-api-access-wx5hv\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.313319 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.313292 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrml\" (UniqueName: \"kubernetes.io/projected/f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9-kube-api-access-ftrml\") pod \"migrator-66df5d496b-7qmpr\" (UID: \"f52fc9a2-fabd-41a9-8626-6a2aae3bf4b9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qmpr" Oct 30 05:53:07.341522 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341471 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-k8s-cni-cncf-io\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.341522 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341497 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-node-log\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.341522 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341514 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7212e5ac-eea5-438a-a162-74821cf38b76-hosts-file\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:53:07.341681 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341529 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cnibin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.341681 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341553 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a5d5930-3046-4003-bce1-4716ab70ef86-host-slash\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:53:07.341681 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341572 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-k8s-cni-cncf-io\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.341681 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341601 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a5d5930-3046-4003-bce1-4716ab70ef86-host-slash\") pod \"iptables-alerter-wmrrw\" (UID: \"8a5d5930-3046-4003-bce1-4716ab70ef86\") " pod="openshift-network-operator/iptables-alerter-wmrrw" Oct 30 05:53:07.341681 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341601 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7212e5ac-eea5-438a-a162-74821cf38b76-hosts-file\") pod \"node-resolver-dz874\" (UID: \"7212e5ac-eea5-438a-a162-74821cf38b76\") " pod="openshift-dns/node-resolver-dz874" Oct 30 05:53:07.341681 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341620 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-system-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.341681 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341662 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.341681 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341622 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-node-log\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341685 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341624 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-cnibin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341703 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-dbus\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341709 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-system-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341728 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341753 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-socket-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341771 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-dbus\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341778 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-registration-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.341783 5001 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341823 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-registration-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341824 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341842 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-socket-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.341850 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:07.841834972 +0000 UTC m=+1.837791960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341897 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341944 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-multus-certs\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.341962 5001 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341968 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-sys-fs\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341982 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-multus-certs\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341993 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-kubelet\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.341973 5001 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.342022 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:53:07.842006955 +0000 UTC m=+1.837963957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : secret "metrics-daemon-secret" not found Oct 30 05:53:07.342026 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342041 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-kubelet\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342055 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-sys-fs\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342087 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysconfig\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342114 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-bin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342138 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-hostroot\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342143 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysconfig\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342165 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342186 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-hostroot\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342193 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342192 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-bin\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342230 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.342236 5001 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342265 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.342275 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls podName:42155cb9-5b97-4aa0-8a18-70af60804e34 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:07.84226137 +0000 UTC m=+1.838218362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gkktj" (UID: "42155cb9-5b97-4aa0-8a18-70af60804e34") : secret "cluster-monitoring-operator-tls" not found Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342318 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-etc-kubernetes\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342338 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342343 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.342373 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:07.84235813 +0000 UTC m=+1.838315137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : configmap references non-existent config key: service-ca.crt Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342362 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a792a513-44ef-43d1-a48c-e17d8f939042-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342390 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-netns\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342387 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-etc-kubernetes\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342413 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-systemd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342414 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342432 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-netns\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.342858 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342374 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.342429 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a792a513-44ef-43d1-a48c-e17d8f939042-crio-socket\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343036 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a792a513-44ef-43d1-a48c-e17d8f939042-crio-socket\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343076 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-system-cni-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343122 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-bin\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343187 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-bin\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343238 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-systemd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343281 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-system-cni-dir\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343299 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-run\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343402 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-run\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343451 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-kubernetes\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343485 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-modprobe-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343510 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-host\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343540 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-etc-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343571 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343602 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-netd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343631 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343666 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-var-lib-kubelet\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343700 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-device-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343732 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-systemd\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343765 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-systemd-units\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343795 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343833 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:07.343903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343879 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-lib-modules\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343918 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-log-socket\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343956 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-kubelet\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.343987 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-sys\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344020 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-netns\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344050 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-multus\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344085 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-ovn\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344117 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-conf\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344154 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-conf-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344188 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-slash\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344256 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344290 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344337 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344371 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cnibin\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344401 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-socket-dir-parent\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344433 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-var-lib-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344461 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ce5e27-0a29-46db-a45d-94dd98cbe059-host\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344481 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-kubelet-config\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344501 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344510 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-log-socket\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344521 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344571 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344691 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-kubernetes\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344700 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-sysctl-conf\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344729 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-ovn\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344753 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-host\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344761 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-modprobe-d\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344790 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-conf-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344804 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-etc-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.344862 5001 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:53:07.344872 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.344870 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-cni-netd\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.344910 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert podName:d6d3694c-d99c-43d1-be3f-6df16717e1de nodeName:}" failed. No retries permitted until 2026-10-30 05:53:07.844899674 +0000 UTC m=+1.840856662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5twq8" (UID: "d6d3694c-d99c-43d1-be3f-6df16717e1de") : secret "networking-console-plugin-cert" not found Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.345415 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7ce5e27-0a29-46db-a45d-94dd98cbe059-host\") pod \"node-ca-l652g\" (UID: \"f7ce5e27-0a29-46db-a45d-94dd98cbe059\") " pod="openshift-image-registry/node-ca-l652g" Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.345481 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6988ac-1b54-4229-9424-aa8e28cbb6c6-cnibin\") pod \"multus-additional-cni-plugins-chl67\" (UID: \"de6988ac-1b54-4229-9424-aa8e28cbb6c6\") " pod="openshift-multus/multus-additional-cni-plugins-chl67" Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.345543 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-socket-dir-parent\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.345590 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-var-lib-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.345701 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-lib-modules\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.345754 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-var-lib-kubelet\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.345808 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5e82585-9fee-4f52-a796-ef82b220e6f6-device-dir\") pod \"aws-ebs-csi-driver-node-c7rmz\" (UID: \"b5e82585-9fee-4f52-a796-ef82b220e6f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7rmz" Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.345874 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-etc-systemd\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.345921 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-systemd-units\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.345969 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.346058 5001 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:07.346121 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.346116 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls podName:800e718b-7419-4df9-836a-cea3686ceda9 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:07.846094152 +0000 UTC m=+1.842051155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls") pod "dns-default-5gv9v" (UID: "800e718b-7419-4df9-836a-cea3686ceda9") : secret "dns-default-metrics-tls" not found Oct 30 05:53:07.346693 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.346162 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-run-netns\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.346693 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.346209 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-kubelet\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.346693 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.346256 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d-sys\") pod \"tuned-85kft\" (UID: \"ff6980f6-e2b4-4e8c-93d9-d1eb30b5b45d\") " pod="openshift-cluster-node-tuning-operator/tuned-85kft" Oct 30 05:53:07.346693 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.346345 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-multus-cni-dir\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.346693 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.346401 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d44df296-4186-4c40-81de-1ed06c15955e-kubelet-config\") pod \"global-pull-secret-syncer-bnv8q\" (UID: \"d44df296-4186-4c40-81de-1ed06c15955e\") " pod="kube-system/global-pull-secret-syncer-bnv8q" Oct 30 05:53:07.346693 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.346460 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.346693 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.346510 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-host-slash\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.346693 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.346556 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11393f29-8fd7-4a36-b3df-03f0992fc7ad-host-var-lib-cni-multus\") pod \"multus-86c7b\" (UID: \"11393f29-8fd7-4a36-b3df-03f0992fc7ad\") " pod="openshift-multus/multus-86c7b" Oct 30 05:53:07.346693 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.346605 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1d67bea-af2e-44ad-ab9b-01123c2889fc-run-openvswitch\") pod \"ovnkube-node-zzk85\" (UID: \"f1d67bea-af2e-44ad-ab9b-01123c2889fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:07.347713 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.347685 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84346da8-55f2-4344-b7d0-dbfb2996f2e7-metrics-certs\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.348085 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.348064 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-glfjr\" (UID: \"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:53:07.349141 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.349119 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\") pod \"image-registry-7b9c9cf4b4-89zvd\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.349729 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.349711 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\") pod \"image-registry-7f9d7c8b8f-gxn9z\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.376742 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.376714 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-mjhts_6c72f864-73b2-48b6-ba6c-fe16323a2d18/migrator/0.log" Oct 30 05:53:07.413875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.413851 5001 scope.go:122] "RemoveContainer" containerID="a1b0e58f3452ccffe96b15479c7192f31726ad35f4c7c369ea791c73d61053fe" Oct 30 05:53:07.413875 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.413873 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" Oct 30 05:53:07.415869 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.415845 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:07.423637 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.423621 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:07.643726 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.643692 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-7chrf_67fb2af9-df3c-41a8-997c-e16de0871a3e/kube-storage-version-migrator-operator/0.log" Oct 30 05:53:07.795051 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.795016 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr"] Oct 30 05:53:07.832788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.832760 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b9c9cf4b4-89zvd"] Oct 30 05:53:07.834446 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.834425 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z"] Oct 30 05:53:07.839823 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:07.839797 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00361e44_6e15_460e_88e7_1fd0398406e5.slice/crio-e6b0d34a41c9abf63e904f2613bf2052bb8c638d0582cc4dac27cd3272480982 WatchSource:0}: Error finding container e6b0d34a41c9abf63e904f2613bf2052bb8c638d0582cc4dac27cd3272480982: Status 404 returned error can't find the container with id e6b0d34a41c9abf63e904f2613bf2052bb8c638d0582cc4dac27cd3272480982 Oct 30 05:53:07.840519 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:07.840498 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184d6270_19ed_4e6b_80c6_9734584b82a1.slice/crio-78ca71ef31aa8464ad98cecd1551427078ab58a495f5cfef6f417d398de728f2 WatchSource:0}: Error finding container 78ca71ef31aa8464ad98cecd1551427078ab58a495f5cfef6f417d398de728f2: Status 404 returned error can't find the container with id 78ca71ef31aa8464ad98cecd1551427078ab58a495f5cfef6f417d398de728f2 Oct 30 05:53:07.851237 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.851215 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:07.851343 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.851254 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:07.851343 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.851286 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:53:07.851343 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.851331 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:07.851501 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.851369 5001 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:07.851501 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.851381 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:07.851501 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.851421 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:08.85140419 +0000 UTC m=+2.847361179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:07.851501 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.851421 5001 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:53:07.851501 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.851456 5001 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:07.851501 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.851455 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:07.851501 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.851481 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:53:08.851464493 +0000 UTC m=+2.847421501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : secret "metrics-daemon-secret" not found Oct 30 05:53:07.851720 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.851506 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls podName:800e718b-7419-4df9-836a-cea3686ceda9 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:08.851495263 +0000 UTC m=+2.847452270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls") pod "dns-default-5gv9v" (UID: "800e718b-7419-4df9-836a-cea3686ceda9") : secret "dns-default-metrics-tls" not found Oct 30 05:53:07.851720 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.851518 5001 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:53:07.851720 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.851551 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert podName:d6d3694c-d99c-43d1-be3f-6df16717e1de nodeName:}" failed. No retries permitted until 2026-10-30 05:53:08.851540083 +0000 UTC m=+2.847497093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5twq8" (UID: "d6d3694c-d99c-43d1-be3f-6df16717e1de") : secret "networking-console-plugin-cert" not found Oct 30 05:53:07.851720 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:07.851606 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:08.851595669 +0000 UTC m=+2.847552698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : configmap references non-existent config key: service-ca.crt Oct 30 05:53:07.853852 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.853836 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42155cb9-5b97-4aa0-8a18-70af60804e34-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gkktj\" (UID: \"42155cb9-5b97-4aa0-8a18-70af60804e34\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:53:07.873959 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:07.873944 5001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:53:08.014806 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.014767 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" Oct 30 05:53:08.184415 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.184295 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" event={"ID":"184d6270-19ed-4e6b-80c6-9734584b82a1","Type":"ContainerStarted","Data":"17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4"} Oct 30 05:53:08.184415 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.184410 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" event={"ID":"184d6270-19ed-4e6b-80c6-9734584b82a1","Type":"ContainerStarted","Data":"78ca71ef31aa8464ad98cecd1551427078ab58a495f5cfef6f417d398de728f2"} Oct 30 05:53:08.185699 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.185675 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" event={"ID":"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef","Type":"ContainerStarted","Data":"2f5cd60ece1464d4b9125e80906dcb09bb8b43f4e7d339fa7bd52fbfa8980ef3"} Oct 30 05:53:08.187266 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.187245 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" event={"ID":"00361e44-6e15-460e-88e7-1fd0398406e5","Type":"ContainerStarted","Data":"5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df"} Oct 30 05:53:08.187384 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.187283 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" event={"ID":"00361e44-6e15-460e-88e7-1fd0398406e5","Type":"ContainerStarted","Data":"e6b0d34a41c9abf63e904f2613bf2052bb8c638d0582cc4dac27cd3272480982"} Oct 30 05:53:08.189293 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.189272 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 05:53:08.189415 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.189396 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" event={"ID":"cb9c2d4d-67e5-4c4d-8294-86ae9050232a","Type":"ContainerStarted","Data":"7fc2a64ae5e91a2992381fb9cc5445537af2bc6cdd61eedbf4529b900ebae90a"} Oct 30 05:53:08.336636 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.336606 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj"] Oct 30 05:53:08.342641 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:08.342617 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42155cb9_5b97_4aa0_8a18_70af60804e34.slice/crio-d21ac94599ce9a421263d17a7b5d418ab3c7cf26f08e1b0a3ccca4d854c1bdea WatchSource:0}: Error finding container d21ac94599ce9a421263d17a7b5d418ab3c7cf26f08e1b0a3ccca4d854c1bdea: Status 404 returned error can't find the container with id d21ac94599ce9a421263d17a7b5d418ab3c7cf26f08e1b0a3ccca4d854c1bdea Oct 30 05:53:08.863489 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.863446 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:08.863667 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.863499 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:08.863667 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.863539 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:08.863667 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.863580 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:08.863667 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:08.863618 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:08.863884 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:08.863738 5001 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:53:08.863884 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:08.863799 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert podName:d6d3694c-d99c-43d1-be3f-6df16717e1de nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.863780838 +0000 UTC m=+4.859737830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5twq8" (UID: "d6d3694c-d99c-43d1-be3f-6df16717e1de") : secret "networking-console-plugin-cert" not found Oct 30 05:53:08.864177 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:08.864159 5001 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:08.864238 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:08.864207 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.864195559 +0000 UTC m=+4.860152547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:08.864327 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:08.864251 5001 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:53:08.864327 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:08.864273 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.864266703 +0000 UTC m=+4.860223690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : secret "metrics-daemon-secret" not found Oct 30 05:53:08.864424 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:08.864357 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.864347152 +0000 UTC m=+4.860304143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : configmap references non-existent config key: service-ca.crt Oct 30 05:53:08.864424 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:08.864416 5001 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:08.864530 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:08.864447 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls podName:800e718b-7419-4df9-836a-cea3686ceda9 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.864435457 +0000 UTC m=+4.860392445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls") pod "dns-default-5gv9v" (UID: "800e718b-7419-4df9-836a-cea3686ceda9") : secret "dns-default-metrics-tls" not found Oct 30 05:53:09.195934 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:09.195835 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" event={"ID":"42155cb9-5b97-4aa0-8a18-70af60804e34","Type":"ContainerStarted","Data":"d21ac94599ce9a421263d17a7b5d418ab3c7cf26f08e1b0a3ccca4d854c1bdea"} Oct 30 05:53:09.195934 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:09.195897 5001 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:53:09.196557 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:09.195897 5001 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:53:09.599622 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:09.599598 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:10.200094 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:10.200054 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gkktj" event={"ID":"42155cb9-5b97-4aa0-8a18-70af60804e34","Type":"ContainerStarted","Data":"ca8cb0b205600f0bcccc80f10a3af0210b7bc804e845db27afb7193840a93649"} Oct 30 05:53:10.201611 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:10.201592 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:10.201958 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:10.201938 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" event={"ID":"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef","Type":"ContainerStarted","Data":"d3d1c808d5605a0dc1b50fb43412c9b2e2b7f99cd4c4d38d54a1e43ced438f5b"} Oct 30 05:53:10.202013 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:10.201980 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-glfjr" event={"ID":"6aa7a82d-6411-4e4b-9bfe-91c37faaa7ef","Type":"ContainerStarted","Data":"999fdfcc666173529f3e80e50af55407c9c4d090307710058e597dd79a23213a"} Oct 30 05:53:10.882544 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:10.882515 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:10.882728 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:10.882557 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:10.882728 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:10.882584 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:10.882728 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:10.882615 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:10.882728 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:10.882644 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:10.882728 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:10.882675 5001 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:53:10.882982 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:10.882734 5001 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:53:10.882982 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:10.882743 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:53:14.88272651 +0000 UTC m=+8.878683498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : secret "metrics-daemon-secret" not found Oct 30 05:53:10.882982 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:10.882676 5001 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:10.882982 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:10.882762 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle podName:84346da8-55f2-4344-b7d0-dbfb2996f2e7 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:14.882749775 +0000 UTC m=+8.878706763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle") pod "router-default-5b994dfc85-2v9ww" (UID: "84346da8-55f2-4344-b7d0-dbfb2996f2e7") : configmap references non-existent config key: service-ca.crt Oct 30 05:53:10.882982 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:10.882782 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:14.88277509 +0000 UTC m=+8.878732078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:10.882982 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:10.882793 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert podName:d6d3694c-d99c-43d1-be3f-6df16717e1de nodeName:}" failed. No retries permitted until 2026-10-30 05:53:14.882787543 +0000 UTC m=+8.878744531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5twq8" (UID: "d6d3694c-d99c-43d1-be3f-6df16717e1de") : secret "networking-console-plugin-cert" not found Oct 30 05:53:10.882982 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:10.882800 5001 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:10.882982 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:10.882910 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls podName:800e718b-7419-4df9-836a-cea3686ceda9 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:14.882890424 +0000 UTC m=+8.878847446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls") pod "dns-default-5gv9v" (UID: "800e718b-7419-4df9-836a-cea3686ceda9") : secret "dns-default-metrics-tls" not found Oct 30 05:53:11.207462 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:11.207382 5001 generic.go:356] "Generic (PLEG): container finished" podID="8f29d013-5eaf-42aa-860a-ba18193d7807" containerID="428158fcd4240a908e4f2817841dfc060be33bea1c07c2c6a40c3a7666541d8d" exitCode=0 Oct 30 05:53:11.207462 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:11.207452 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lq9h9" event={"ID":"8f29d013-5eaf-42aa-860a-ba18193d7807","Type":"ContainerDied","Data":"428158fcd4240a908e4f2817841dfc060be33bea1c07c2c6a40c3a7666541d8d"} Oct 30 05:53:11.207931 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:11.207916 5001 scope.go:122] "RemoveContainer" containerID="428158fcd4240a908e4f2817841dfc060be33bea1c07c2c6a40c3a7666541d8d" Oct 30 05:53:11.707424 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:11.707392 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:53:12.212252 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:12.212215 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lq9h9" event={"ID":"8f29d013-5eaf-42aa-860a-ba18193d7807","Type":"ContainerStarted","Data":"1b9b7dba644c634b5d99c8f14e872806f104ea9e0f579bf91594c9b546803074"} Oct 30 05:53:12.847721 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:12.847690 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:12.847897 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:12.847885 5001 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:53:13.125320 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:13.125273 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:13.125485 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:13.125431 5001 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:53:13.130010 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:13.129988 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7ztmq" Oct 30 05:53:13.667944 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:13.667916 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzk85" Oct 30 05:53:14.919682 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:14.919647 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:14.920115 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:14.919690 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:14.920115 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:14.919779 5001 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:53:14.920115 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:14.919782 5001 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:14.920115 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:14.919810 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:14.920115 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:14.919836 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs podName:cad7f8eb-8604-4b8b-a921-38b9d5c281db nodeName:}" failed. No retries permitted until 2026-10-30 05:53:22.919824453 +0000 UTC m=+16.915781473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs") pod "network-metrics-daemon-r5rsl" (UID: "cad7f8eb-8604-4b8b-a921-38b9d5c281db") : secret "metrics-daemon-secret" not found Oct 30 05:53:14.920115 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:14.919851 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls podName:a792a513-44ef-43d1-a48c-e17d8f939042 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:22.919845491 +0000 UTC m=+16.915802479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtvjb" (UID: "a792a513-44ef-43d1-a48c-e17d8f939042") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:14.920115 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:14.919913 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:14.920115 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:14.919950 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:14.920115 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:14.920040 5001 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:53:14.920115 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:14.920081 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert podName:d6d3694c-d99c-43d1-be3f-6df16717e1de nodeName:}" failed. No retries permitted until 2026-10-30 05:53:22.920067455 +0000 UTC m=+16.916024472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5twq8" (UID: "d6d3694c-d99c-43d1-be3f-6df16717e1de") : secret "networking-console-plugin-cert" not found Oct 30 05:53:14.920478 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:14.920353 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84346da8-55f2-4344-b7d0-dbfb2996f2e7-service-ca-bundle\") pod \"router-default-5b994dfc85-2v9ww\" (UID: \"84346da8-55f2-4344-b7d0-dbfb2996f2e7\") " pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:14.922471 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:14.922453 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/800e718b-7419-4df9-836a-cea3686ceda9-metrics-tls\") pod \"dns-default-5gv9v\" (UID: \"800e718b-7419-4df9-836a-cea3686ceda9\") " pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:14.925420 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:14.925397 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:15.213606 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:15.213526 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:15.228490 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:15.223834 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5gv9v"] Oct 30 05:53:15.231824 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:15.231796 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800e718b_7419_4df9_836a_cea3686ceda9.slice/crio-455ec17b9d019c128c0a49ef653c99a0bd0b3a3d22fcdc574c3defbf144c7a5a WatchSource:0}: Error finding container 455ec17b9d019c128c0a49ef653c99a0bd0b3a3d22fcdc574c3defbf144c7a5a: Status 404 returned error can't find the container with id 455ec17b9d019c128c0a49ef653c99a0bd0b3a3d22fcdc574c3defbf144c7a5a Oct 30 05:53:15.531915 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:15.531835 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b994dfc85-2v9ww"] Oct 30 05:53:15.536964 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:15.536937 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84346da8_55f2_4344_b7d0_dbfb2996f2e7.slice/crio-507d24c62197e4483f903bf702137825e9e6292166d147c9fd40bbef5aa4baef WatchSource:0}: Error finding container 507d24c62197e4483f903bf702137825e9e6292166d147c9fd40bbef5aa4baef: Status 404 returned error can't find the container with id 507d24c62197e4483f903bf702137825e9e6292166d147c9fd40bbef5aa4baef Oct 30 05:53:16.234155 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:16.234122 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5gv9v" event={"ID":"800e718b-7419-4df9-836a-cea3686ceda9","Type":"ContainerStarted","Data":"455ec17b9d019c128c0a49ef653c99a0bd0b3a3d22fcdc574c3defbf144c7a5a"} Oct 30 05:53:16.235991 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:16.235959 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b994dfc85-2v9ww" event={"ID":"84346da8-55f2-4344-b7d0-dbfb2996f2e7","Type":"ContainerStarted","Data":"3546ec9e8c001f809dd9c7ceb32259974d50418e7f6720a369aa684c064038f9"} Oct 30 05:53:16.236138 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:16.236007 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b994dfc85-2v9ww" event={"ID":"84346da8-55f2-4344-b7d0-dbfb2996f2e7","Type":"ContainerStarted","Data":"507d24c62197e4483f903bf702137825e9e6292166d147c9fd40bbef5aa4baef"} Oct 30 05:53:16.292906 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:16.292875 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:17.214247 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:17.214209 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:17.216881 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:17.216859 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:17.240896 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:17.240870 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5gv9v" event={"ID":"800e718b-7419-4df9-836a-cea3686ceda9","Type":"ContainerStarted","Data":"3908cb68b20de663833fd4231bf2a77434c602713f26d76e63a29a23c0c73867"} Oct 30 05:53:17.241255 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:17.240917 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5gv9v" event={"ID":"800e718b-7419-4df9-836a-cea3686ceda9","Type":"ContainerStarted","Data":"c4623ffeb06975f9a6e2bf0ee1fe68b4f940f0871f55a619b091ac6d6a4c7c39"} Oct 30 05:53:17.241255 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:17.240971 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:17.242738 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:17.242718 5001 generic.go:356] "Generic (PLEG): container finished" podID="8f29d013-5eaf-42aa-860a-ba18193d7807" containerID="1b9b7dba644c634b5d99c8f14e872806f104ea9e0f579bf91594c9b546803074" exitCode=0 Oct 30 05:53:17.242834 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:17.242789 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lq9h9" event={"ID":"8f29d013-5eaf-42aa-860a-ba18193d7807","Type":"ContainerDied","Data":"1b9b7dba644c634b5d99c8f14e872806f104ea9e0f579bf91594c9b546803074"} Oct 30 05:53:17.242834 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:17.242828 5001 scope.go:122] "RemoveContainer" containerID="428158fcd4240a908e4f2817841dfc060be33bea1c07c2c6a40c3a7666541d8d" Oct 30 05:53:17.243145 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:17.243129 5001 scope.go:122] "RemoveContainer" containerID="1b9b7dba644c634b5d99c8f14e872806f104ea9e0f579bf91594c9b546803074" Oct 30 05:53:17.243350 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:17.243331 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-lq9h9_openshift-insights(8f29d013-5eaf-42aa-860a-ba18193d7807)\"" pod="openshift-insights/insights-operator-7c4986d449-lq9h9" podUID="8f29d013-5eaf-42aa-860a-ba18193d7807" Oct 30 05:53:17.366029 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:17.365996 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:17.367197 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:17.367179 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5b994dfc85-2v9ww" Oct 30 05:53:19.991249 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:19.991215 5001 patch_prober.go:29] interesting pod/image-registry-7f9d7c8b8f-gxn9z container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 05:53:19.991655 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:19.991283 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" podUID="184d6270-19ed-4e6b-80c6-9734584b82a1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:53:22.726656 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.726620 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-pdm7b-xpzr9"] Oct 30 05:53:22.729466 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.729448 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.760962 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.760935 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pdm7b-xpzr9"] Oct 30 05:53:22.786370 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.786339 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-service-ca-bundle\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.786496 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.786406 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnwn\" (UniqueName: \"kubernetes.io/projected/e955d2b2-7e74-4bd9-a546-52c533182da7-kube-api-access-xgnwn\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.786496 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.786482 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e955d2b2-7e74-4bd9-a546-52c533182da7-archives-path\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.786575 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.786525 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-trusted-ca-bundle\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.826972 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.826935 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z"] Oct 30 05:53:22.832954 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.832922 5001 patch_prober.go:29] interesting pod/image-registry-7f9d7c8b8f-gxn9z container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 05:53:22.833109 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.832970 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" podUID="184d6270-19ed-4e6b-80c6-9734584b82a1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:53:22.887767 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.887732 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-service-ca-bundle\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.887951 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.887798 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnwn\" (UniqueName: \"kubernetes.io/projected/e955d2b2-7e74-4bd9-a546-52c533182da7-kube-api-access-xgnwn\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.887951 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.887911 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e955d2b2-7e74-4bd9-a546-52c533182da7-archives-path\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.887951 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.887947 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-trusted-ca-bundle\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.888836 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.888812 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e955d2b2-7e74-4bd9-a546-52c533182da7-archives-path\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.889075 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.889056 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-service-ca-bundle\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.889171 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.889155 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-trusted-ca-bundle\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.915163 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.915139 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnwn\" (UniqueName: \"kubernetes.io/projected/e955d2b2-7e74-4bd9-a546-52c533182da7-kube-api-access-xgnwn\") pod \"periodic-gathering-pdm7b-xpzr9\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:22.988594 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.988516 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:22.988594 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.988576 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:22.988837 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.988602 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:22.991055 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.991026 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad7f8eb-8604-4b8b-a921-38b9d5c281db-metrics-certs\") pod \"network-metrics-daemon-r5rsl\" (UID: \"cad7f8eb-8604-4b8b-a921-38b9d5c281db\") " pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:22.991172 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.991060 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a792a513-44ef-43d1-a48c-e17d8f939042-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtvjb\" (UID: \"a792a513-44ef-43d1-a48c-e17d8f939042\") " pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:22.991209 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:22.991173 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6d3694c-d99c-43d1-be3f-6df16717e1de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5twq8\" (UID: \"d6d3694c-d99c-43d1-be3f-6df16717e1de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:23.017175 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:23.017146 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" Oct 30 05:53:23.019387 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:23.019370 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5rsl" Oct 30 05:53:23.027910 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:23.027891 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtvjb" Oct 30 05:53:23.039228 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:23.039201 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:23.324605 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:23.324534 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5rsl"] Oct 30 05:53:23.330842 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:23.330815 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad7f8eb_8604_4b8b_a921_38b9d5c281db.slice/crio-1ae680cbffd0777bdccca137e8e6d90f701da40a39c7a47ca04890b395e437b2 WatchSource:0}: Error finding container 1ae680cbffd0777bdccca137e8e6d90f701da40a39c7a47ca04890b395e437b2: Status 404 returned error can't find the container with id 1ae680cbffd0777bdccca137e8e6d90f701da40a39c7a47ca04890b395e437b2 Oct 30 05:53:23.337917 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:23.337897 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5twq8"] Oct 30 05:53:23.342463 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:23.342441 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d3694c_d99c_43d1_be3f_6df16717e1de.slice/crio-eecf21fdbfa01a8aa463f0767f6d30dde96689155510fe5e8a446fa044c26d81 WatchSource:0}: Error finding container eecf21fdbfa01a8aa463f0767f6d30dde96689155510fe5e8a446fa044c26d81: Status 404 returned error can't find the container with id eecf21fdbfa01a8aa463f0767f6d30dde96689155510fe5e8a446fa044c26d81 Oct 30 05:53:23.353586 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:23.353562 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dtvjb"] Oct 30 05:53:23.358638 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:23.358617 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda792a513_44ef_43d1_a48c_e17d8f939042.slice/crio-56600b6e2e7fe5c940c30ac869f3327d3abe749ce3c0c30d999c2fccf44a907b WatchSource:0}: Error finding container 56600b6e2e7fe5c940c30ac869f3327d3abe749ce3c0c30d999c2fccf44a907b: Status 404 returned error can't find the container with id 56600b6e2e7fe5c940c30ac869f3327d3abe749ce3c0c30d999c2fccf44a907b Oct 30 05:53:23.370360 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:23.370334 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pdm7b-xpzr9"] Oct 30 05:53:23.375218 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:23.375197 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode955d2b2_7e74_4bd9_a546_52c533182da7.slice/crio-789c360fb949777afe8cd2e0685ed50df209682fd6afb591a90763d434a1f712 WatchSource:0}: Error finding container 789c360fb949777afe8cd2e0685ed50df209682fd6afb591a90763d434a1f712: Status 404 returned error can't find the container with id 789c360fb949777afe8cd2e0685ed50df209682fd6afb591a90763d434a1f712 Oct 30 05:53:23.887190 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:23.887069 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cq5xz" Oct 30 05:53:24.272235 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:24.272075 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" event={"ID":"d6d3694c-d99c-43d1-be3f-6df16717e1de","Type":"ContainerStarted","Data":"eecf21fdbfa01a8aa463f0767f6d30dde96689155510fe5e8a446fa044c26d81"} Oct 30 05:53:24.274088 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:24.274058 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" event={"ID":"e955d2b2-7e74-4bd9-a546-52c533182da7","Type":"ContainerStarted","Data":"83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82"} Oct 30 05:53:24.274214 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:24.274113 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" event={"ID":"e955d2b2-7e74-4bd9-a546-52c533182da7","Type":"ContainerStarted","Data":"789c360fb949777afe8cd2e0685ed50df209682fd6afb591a90763d434a1f712"} Oct 30 05:53:24.276530 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:24.276503 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtvjb" event={"ID":"a792a513-44ef-43d1-a48c-e17d8f939042","Type":"ContainerStarted","Data":"0e89afc5384692b338071644b05aeeb5465fddc9090e50b15654d557ec7ba038"} Oct 30 05:53:24.276647 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:24.276560 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtvjb" event={"ID":"a792a513-44ef-43d1-a48c-e17d8f939042","Type":"ContainerStarted","Data":"2dafb10bc18ec5db86a9a956cbb7cff2498b85fd85efa04f2ff76b44063818fb"} Oct 30 05:53:24.276647 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:24.276588 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtvjb" event={"ID":"a792a513-44ef-43d1-a48c-e17d8f939042","Type":"ContainerStarted","Data":"56600b6e2e7fe5c940c30ac869f3327d3abe749ce3c0c30d999c2fccf44a907b"} Oct 30 05:53:24.277994 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:24.277944 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5rsl" event={"ID":"cad7f8eb-8604-4b8b-a921-38b9d5c281db","Type":"ContainerStarted","Data":"1ae680cbffd0777bdccca137e8e6d90f701da40a39c7a47ca04890b395e437b2"} Oct 30 05:53:24.619053 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:24.619026 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5gv9v_800e718b-7419-4df9-836a-cea3686ceda9/dns/0.log" Oct 30 05:53:24.816959 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:24.816880 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5gv9v_800e718b-7419-4df9-836a-cea3686ceda9/kube-rbac-proxy/0.log" Oct 30 05:53:25.284039 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:25.283994 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5rsl" event={"ID":"cad7f8eb-8604-4b8b-a921-38b9d5c281db","Type":"ContainerStarted","Data":"7516a0a0ce006c6e6c217b70cd93ec47312d7e6fe8da74171d7197f8c3087064"} Oct 30 05:53:25.284517 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:25.284060 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5rsl" event={"ID":"cad7f8eb-8604-4b8b-a921-38b9d5c281db","Type":"ContainerStarted","Data":"d852fe53ee85b3de44bed8b4846e4b86a0e35bf2c4a324ec1dbcc4811af3e942"} Oct 30 05:53:25.285903 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:25.285873 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5twq8" event={"ID":"d6d3694c-d99c-43d1-be3f-6df16717e1de","Type":"ContainerStarted","Data":"e86589235a1d8564268e46953b46911a862b7df7d9a929c7ef2bedc681449087"} Oct 30 05:53:25.331336 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:25.331253 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" podStartSLOduration=3.331237787 podStartE2EDuration="3.331237787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:24.319918267 +0000 UTC m=+18.315875278" watchObservedRunningTime="2026-10-30 05:53:25.331237787 +0000 UTC m=+19.327194871" Oct 30 05:53:25.616990 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:25.616966 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dz874_7212e5ac-eea5-438a-a162-74821cf38b76/dns-node-resolver/0.log" Oct 30 05:53:25.812786 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:25.812749 5001 patch_prober.go:29] interesting pod/image-registry-7b9c9cf4b4-89zvd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 05:53:25.812943 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:25.812803 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" podUID="00361e44-6e15-460e-88e7-1fd0398406e5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:53:26.217241 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:26.217207 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7b9c9cf4b4-89zvd_00361e44-6e15-460e-88e7-1fd0398406e5/registry/0.log" Oct 30 05:53:26.291962 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:26.291934 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtvjb" event={"ID":"a792a513-44ef-43d1-a48c-e17d8f939042","Type":"ContainerStarted","Data":"be68fb025b25a41075beb63cf6fb64db15200394d1a4022c7b38ce7e416927c1"} Oct 30 05:53:26.297866 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:26.297833 5001 patch_prober.go:29] interesting pod/image-registry-7b9c9cf4b4-89zvd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 05:53:26.298001 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:26.297873 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" podUID="00361e44-6e15-460e-88e7-1fd0398406e5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:53:26.416696 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:26.416667 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f9d7c8b8f-gxn9z_184d6270-19ed-4e6b-80c6-9734584b82a1/registry/0.log" Oct 30 05:53:26.616693 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:26.616669 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l652g_f7ce5e27-0a29-46db-a45d-94dd98cbe059/node-ca/0.log" Oct 30 05:53:27.109605 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:27.109577 5001 scope.go:122] "RemoveContainer" containerID="1b9b7dba644c634b5d99c8f14e872806f104ea9e0f579bf91594c9b546803074" Oct 30 05:53:27.217666 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:27.217633 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b994dfc85-2v9ww_84346da8-55f2-4344-b7d0-dbfb2996f2e7/router/0.log" Oct 30 05:53:27.250033 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:27.250007 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5gv9v" Oct 30 05:53:27.297789 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:27.297757 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-lq9h9" event={"ID":"8f29d013-5eaf-42aa-860a-ba18193d7807","Type":"ContainerStarted","Data":"314b9967f489b36457db1d24caf850db2daca1912b4f308c60c7c4df6666a547"} Oct 30 05:53:27.616826 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:27.616788 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lrww4_93ed3e75-7873-4300-be92-0cff9a0d63a7/serve-healthcheck-canary/0.log" Oct 30 05:53:31.296025 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.295990 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vjstw"] Oct 30 05:53:31.316741 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.316719 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.365631 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.365605 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 05:53:31.365783 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.365677 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 05:53:31.365783 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.365753 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rkzps\"" Oct 30 05:53:31.367023 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.367002 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.367104 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.367042 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.367104 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.367074 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-tls\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.367245 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.367223 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-sys\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.367350 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.367331 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-wtmp\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.367432 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.367368 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-textfile\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.367490 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.367447 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-root\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.367543 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.367498 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-metrics-client-ca\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.367543 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.367515 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x85cm\" (UniqueName: \"kubernetes.io/projected/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-kube-api-access-x85cm\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.379995 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.379976 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 05:53:31.380115 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.380058 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 05:53:31.468572 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468540 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-tls\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.468719 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468578 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-sys\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.468719 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468611 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-wtmp\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.468719 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468636 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-textfile\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.468719 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468660 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-root\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.468719 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468670 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-sys\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.468968 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468724 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-metrics-client-ca\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.468968 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468748 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x85cm\" (UniqueName: \"kubernetes.io/projected/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-kube-api-access-x85cm\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.468968 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468769 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-root\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.468968 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468794 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.468968 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468810 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-wtmp\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.468968 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.468918 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.469218 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.469050 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-textfile\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.469423 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.469404 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.469470 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.469417 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-metrics-client-ca\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.471054 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.471034 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-tls\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.471263 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.471245 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.492788 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.492765 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x85cm\" (UniqueName: \"kubernetes.io/projected/34b1a113-68c8-4b8c-a46f-52b4c63d11e6-kube-api-access-x85cm\") pod \"node-exporter-vjstw\" (UID: \"34b1a113-68c8-4b8c-a46f-52b4c63d11e6\") " pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.665695 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:31.665663 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjstw" Oct 30 05:53:31.700240 ip-10-0-128-31 kubenswrapper[5001]: W1030 05:53:31.700209 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b1a113_68c8_4b8c_a46f_52b4c63d11e6.slice/crio-9a2d24952858f28a5f17a0228b755769a6ded5b24caba65228241c00b6652ecf WatchSource:0}: Error finding container 9a2d24952858f28a5f17a0228b755769a6ded5b24caba65228241c00b6652ecf: Status 404 returned error can't find the container with id 9a2d24952858f28a5f17a0228b755769a6ded5b24caba65228241c00b6652ecf Oct 30 05:53:32.321521 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:32.321487 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjstw" event={"ID":"34b1a113-68c8-4b8c-a46f-52b4c63d11e6","Type":"ContainerStarted","Data":"9a2d24952858f28a5f17a0228b755769a6ded5b24caba65228241c00b6652ecf"} Oct 30 05:53:32.831649 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:32.831625 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:33.326761 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:33.326725 5001 generic.go:356] "Generic (PLEG): container finished" podID="34b1a113-68c8-4b8c-a46f-52b4c63d11e6" containerID="120692d0362b71f232040c5aadb5723b5d1ebe74b9c514cee930bbbfd0cb0350" exitCode=0 Oct 30 05:53:33.327157 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:33.326806 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjstw" event={"ID":"34b1a113-68c8-4b8c-a46f-52b4c63d11e6","Type":"ContainerDied","Data":"120692d0362b71f232040c5aadb5723b5d1ebe74b9c514cee930bbbfd0cb0350"} Oct 30 05:53:34.333455 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:34.333414 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjstw" event={"ID":"34b1a113-68c8-4b8c-a46f-52b4c63d11e6","Type":"ContainerStarted","Data":"e0a6a0c211c937f8d4c58c4aba9a851699dc032dc9e3fd84ed182bab2727950a"} Oct 30 05:53:34.333815 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:34.333474 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjstw" event={"ID":"34b1a113-68c8-4b8c-a46f-52b4c63d11e6","Type":"ContainerStarted","Data":"463485ed8ce58b45abaaae750a803f359d0ab419902d2e3c529b8435dba527d6"} Oct 30 05:53:34.401536 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:34.401475 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vjstw" podStartSLOduration=2.024314355 podStartE2EDuration="3.401458297s" totalImagesPullingTime="1.294639992s" totalInitContainerRuntime="82.50395ms" isStatefulPod=true podCreationTimestamp="2026-10-30 05:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:34.38922814 +0000 UTC m=+28.385185160" watchObservedRunningTime="2026-10-30 05:53:34.401458297 +0000 UTC m=+28.397415309" Oct 30 05:53:36.297843 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:36.297808 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:53:38.264063 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:38.264022 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:53:41.710555 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:41.710524 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xsrdq" Oct 30 05:53:45.547454 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:45.547408 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b9c9cf4b4-89zvd"] Oct 30 05:53:47.848483 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:47.848360 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" podUID="184d6270-19ed-4e6b-80c6-9734584b82a1" containerName="registry" containerID="cri-o://17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4" gracePeriod=30 Oct 30 05:53:48.099857 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.099802 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:48.227023 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.226991 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184d6270-19ed-4e6b-80c6-9734584b82a1-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/184d6270-19ed-4e6b-80c6-9734584b82a1-ca-trust-extracted\") pod \"184d6270-19ed-4e6b-80c6-9734584b82a1\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " Oct 30 05:53:48.227192 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.227050 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-image-registry-private-configuration\") pod \"184d6270-19ed-4e6b-80c6-9734584b82a1\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " Oct 30 05:53:48.227192 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.227079 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\") pod \"184d6270-19ed-4e6b-80c6-9734584b82a1\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " Oct 30 05:53:48.227192 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.227100 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-certificates\") pod \"184d6270-19ed-4e6b-80c6-9734584b82a1\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " Oct 30 05:53:48.227192 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.227127 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-installation-pull-secrets\") pod \"184d6270-19ed-4e6b-80c6-9734584b82a1\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " Oct 30 05:53:48.227192 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.227142 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-trusted-ca\") pod \"184d6270-19ed-4e6b-80c6-9734584b82a1\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " Oct 30 05:53:48.227486 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.227289 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-kube-api-access-krbp7\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-kube-api-access-krbp7\") pod \"184d6270-19ed-4e6b-80c6-9734584b82a1\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " Oct 30 05:53:48.227486 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.227360 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-bound-sa-token\") pod \"184d6270-19ed-4e6b-80c6-9734584b82a1\" (UID: \"184d6270-19ed-4e6b-80c6-9734584b82a1\") " Oct 30 05:53:48.227486 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.227453 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-certificates" pod "184d6270-19ed-4e6b-80c6-9734584b82a1" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:48.227688 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.227668 5001 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-certificates\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:48.228362 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.228332 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-trusted-ca" pod "184d6270-19ed-4e6b-80c6-9734584b82a1" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:48.230022 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.229994 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-bound-sa-token" pod "184d6270-19ed-4e6b-80c6-9734584b82a1" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:48.230022 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.230006 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls" pod "184d6270-19ed-4e6b-80c6-9734584b82a1" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:48.230163 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.230019 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-installation-pull-secrets" pod "184d6270-19ed-4e6b-80c6-9734584b82a1" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:53:48.230225 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.230204 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-image-registry-private-configuration" pod "184d6270-19ed-4e6b-80c6-9734584b82a1" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:53:48.230260 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.230233 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-kube-api-access-krbp7" pod "184d6270-19ed-4e6b-80c6-9734584b82a1" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1"). InnerVolumeSpecName "kube-api-access-krbp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:48.235713 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.235685 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184d6270-19ed-4e6b-80c6-9734584b82a1-ca-trust-extracted" pod "184d6270-19ed-4e6b-80c6-9734584b82a1" (UID: "184d6270-19ed-4e6b-80c6-9734584b82a1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:53:48.328634 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.328599 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krbp7\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-kube-api-access-krbp7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:48.328634 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.328629 5001 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-bound-sa-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:48.328634 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.328639 5001 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/184d6270-19ed-4e6b-80c6-9734584b82a1-ca-trust-extracted\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:48.328849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.328651 5001 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-image-registry-private-configuration\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:48.328849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.328661 5001 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/184d6270-19ed-4e6b-80c6-9734584b82a1-registry-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:48.328849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.328670 5001 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/184d6270-19ed-4e6b-80c6-9734584b82a1-installation-pull-secrets\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:48.328849 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.328679 5001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/184d6270-19ed-4e6b-80c6-9734584b82a1-trusted-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:48.398789 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.398761 5001 generic.go:356] "Generic (PLEG): container finished" podID="184d6270-19ed-4e6b-80c6-9734584b82a1" containerID="17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4" exitCode=0 Oct 30 05:53:48.398916 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.398810 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" Oct 30 05:53:48.398916 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.398828 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" event={"ID":"184d6270-19ed-4e6b-80c6-9734584b82a1","Type":"ContainerDied","Data":"17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4"} Oct 30 05:53:48.398916 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.398867 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z" event={"ID":"184d6270-19ed-4e6b-80c6-9734584b82a1","Type":"ContainerDied","Data":"78ca71ef31aa8464ad98cecd1551427078ab58a495f5cfef6f417d398de728f2"} Oct 30 05:53:48.398916 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.398888 5001 scope.go:122] "RemoveContainer" containerID="17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4" Oct 30 05:53:48.407631 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.407613 5001 scope.go:122] "RemoveContainer" containerID="17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4" Oct 30 05:53:48.407883 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:48.407865 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4\": container with ID starting with 17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4 not found: ID does not exist" containerID="17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4" Oct 30 05:53:48.407942 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.407891 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4"} err="failed to get container status \"17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4\": rpc error: code = NotFound desc = could not find container \"17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4\": container with ID starting with 17481bbcbbd99de1c440045bf8025a11bbb8021a796afa36eda4734870aae3a4 not found: ID does not exist" Oct 30 05:53:48.467100 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.467074 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z"] Oct 30 05:53:48.477837 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:48.477807 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f9d7c8b8f-gxn9z"] Oct 30 05:53:50.114459 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:50.114426 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184d6270-19ed-4e6b-80c6-9734584b82a1" path="/var/lib/kubelet/pods/184d6270-19ed-4e6b-80c6-9734584b82a1/volumes" Oct 30 05:53:55.884509 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:55.884480 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pdm7b-xpzr9"] Oct 30 05:53:55.885299 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:55.884672 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" podUID="e955d2b2-7e74-4bd9-a546-52c533182da7" containerName="insights-gathering" containerID="cri-o://83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82" gracePeriod=30 Oct 30 05:53:55.895840 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:55.895815 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-ghpdk-jn8g9"] Oct 30 05:53:55.896126 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:55.896083 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" podUID="4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" containerName="insights-gathering" containerID="cri-o://65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725" gracePeriod=30 Oct 30 05:53:56.155805 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.155783 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:56.182002 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.181977 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:56.294329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.294273 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e955d2b2-7e74-4bd9-a546-52c533182da7-kube-api-access-xgnwn\" (UniqueName: \"kubernetes.io/projected/e955d2b2-7e74-4bd9-a546-52c533182da7-kube-api-access-xgnwn\") pod \"e955d2b2-7e74-4bd9-a546-52c533182da7\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " Oct 30 05:53:56.294329 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.294327 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-trusted-ca-bundle\") pod \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " Oct 30 05:53:56.294576 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.294362 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-kube-api-access-hgl2n\" (UniqueName: \"kubernetes.io/projected/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-kube-api-access-hgl2n\") pod \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " Oct 30 05:53:56.294576 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.294385 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-service-ca-bundle\") pod \"e955d2b2-7e74-4bd9-a546-52c533182da7\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " Oct 30 05:53:56.294576 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.294407 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-service-ca-bundle\") pod \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " Oct 30 05:53:56.294576 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.294504 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e955d2b2-7e74-4bd9-a546-52c533182da7-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e955d2b2-7e74-4bd9-a546-52c533182da7-archives-path\") pod \"e955d2b2-7e74-4bd9-a546-52c533182da7\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " Oct 30 05:53:56.294576 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.294540 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-archives-path\") pod \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\" (UID: \"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9\") " Oct 30 05:53:56.294576 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.294570 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-trusted-ca-bundle\") pod \"e955d2b2-7e74-4bd9-a546-52c533182da7\" (UID: \"e955d2b2-7e74-4bd9-a546-52c533182da7\") " Oct 30 05:53:56.294866 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.294776 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-service-ca-bundle" pod "e955d2b2-7e74-4bd9-a546-52c533182da7" (UID: "e955d2b2-7e74-4bd9-a546-52c533182da7"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:56.294866 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.294802 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-trusted-ca-bundle" pod "4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" (UID: "4bc92757-4a0b-44d6-9183-9d28c1b0d5e9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:56.295353 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.295327 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-service-ca-bundle" pod "4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" (UID: "4bc92757-4a0b-44d6-9183-9d28c1b0d5e9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:56.295438 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.295358 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-trusted-ca-bundle" pod "e955d2b2-7e74-4bd9-a546-52c533182da7" (UID: "e955d2b2-7e74-4bd9-a546-52c533182da7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:56.295481 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.295449 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e955d2b2-7e74-4bd9-a546-52c533182da7-archives-path" pod "e955d2b2-7e74-4bd9-a546-52c533182da7" (UID: "e955d2b2-7e74-4bd9-a546-52c533182da7"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:53:56.295518 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.295495 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-archives-path" pod "4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" (UID: "4bc92757-4a0b-44d6-9183-9d28c1b0d5e9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:53:56.296710 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.296679 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e955d2b2-7e74-4bd9-a546-52c533182da7-kube-api-access-xgnwn" pod "e955d2b2-7e74-4bd9-a546-52c533182da7" (UID: "e955d2b2-7e74-4bd9-a546-52c533182da7"). InnerVolumeSpecName "kube-api-access-xgnwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:56.296832 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.296817 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-kube-api-access-hgl2n" pod "4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" (UID: "4bc92757-4a0b-44d6-9183-9d28c1b0d5e9"). InnerVolumeSpecName "kube-api-access-hgl2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:56.395576 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.395516 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgnwn\" (UniqueName: \"kubernetes.io/projected/e955d2b2-7e74-4bd9-a546-52c533182da7-kube-api-access-xgnwn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:56.395576 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.395540 5001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-trusted-ca-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:56.395576 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.395549 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgl2n\" (UniqueName: \"kubernetes.io/projected/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-kube-api-access-hgl2n\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:56.395576 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.395579 5001 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-service-ca-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:56.395774 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.395588 5001 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-service-ca-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:56.395774 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.395598 5001 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e955d2b2-7e74-4bd9-a546-52c533182da7-archives-path\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:56.395774 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.395607 5001 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9-archives-path\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:56.395774 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.395614 5001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e955d2b2-7e74-4bd9-a546-52c533182da7-trusted-ca-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:53:56.431737 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.431711 5001 generic.go:356] "Generic (PLEG): container finished" podID="e955d2b2-7e74-4bd9-a546-52c533182da7" containerID="83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82" exitCode=2 Oct 30 05:53:56.431877 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.431757 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" Oct 30 05:53:56.431877 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.431812 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" event={"ID":"e955d2b2-7e74-4bd9-a546-52c533182da7","Type":"ContainerDied","Data":"83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82"} Oct 30 05:53:56.431877 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.431849 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pdm7b-xpzr9" event={"ID":"e955d2b2-7e74-4bd9-a546-52c533182da7","Type":"ContainerDied","Data":"789c360fb949777afe8cd2e0685ed50df209682fd6afb591a90763d434a1f712"} Oct 30 05:53:56.431877 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.431867 5001 scope.go:122] "RemoveContainer" containerID="83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82" Oct 30 05:53:56.433127 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.433105 5001 generic.go:356] "Generic (PLEG): container finished" podID="4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" containerID="65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725" exitCode=2 Oct 30 05:53:56.433210 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.433141 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" Oct 30 05:53:56.433210 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.433155 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" event={"ID":"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9","Type":"ContainerDied","Data":"65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725"} Oct 30 05:53:56.433210 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.433181 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ghpdk-jn8g9" event={"ID":"4bc92757-4a0b-44d6-9183-9d28c1b0d5e9","Type":"ContainerDied","Data":"7e40cc373f4722e46eeee669f2b7f968458827ab0e49b430efa82d3190db4463"} Oct 30 05:53:56.441288 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.441256 5001 scope.go:122] "RemoveContainer" containerID="83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82" Oct 30 05:53:56.441621 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:56.441602 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82\": container with ID starting with 83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82 not found: ID does not exist" containerID="83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82" Oct 30 05:53:56.441708 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.441627 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82"} err="failed to get container status \"83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82\": rpc error: code = NotFound desc = could not find container \"83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82\": container with ID starting with 83aedce7c6b7a49e01097fe2d7a4ce1f6959efbf0527520359ca28518698ba82 not found: ID does not exist" Oct 30 05:53:56.441708 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.441649 5001 scope.go:122] "RemoveContainer" containerID="65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725" Oct 30 05:53:56.450016 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.450002 5001 scope.go:122] "RemoveContainer" containerID="65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725" Oct 30 05:53:56.450274 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:53:56.450258 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725\": container with ID starting with 65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725 not found: ID does not exist" containerID="65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725" Oct 30 05:53:56.450339 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.450281 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725"} err="failed to get container status \"65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725\": rpc error: code = NotFound desc = could not find container \"65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725\": container with ID starting with 65080f52afcecbbfe0078e50bec0b5b4823c044c5e0d685b9a5bdafc63821725 not found: ID does not exist" Oct 30 05:53:56.484633 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.484609 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-ghpdk-jn8g9"] Oct 30 05:53:56.489786 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.489759 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-ghpdk-jn8g9"] Oct 30 05:53:56.532186 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.532161 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pdm7b-xpzr9"] Oct 30 05:53:56.540625 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:56.540602 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-pdm7b-xpzr9"] Oct 30 05:53:58.114811 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:58.114778 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" path="/var/lib/kubelet/pods/4bc92757-4a0b-44d6-9183-9d28c1b0d5e9/volumes" Oct 30 05:53:58.115190 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:53:58.115144 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e955d2b2-7e74-4bd9-a546-52c533182da7" path="/var/lib/kubelet/pods/e955d2b2-7e74-4bd9-a546-52c533182da7/volumes" Oct 30 05:54:08.265645 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:54:08.265604 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:54:10.567664 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.567611 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" podUID="00361e44-6e15-460e-88e7-1fd0398406e5" containerName="registry" containerID="cri-o://5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df" gracePeriod=30 Oct 30 05:54:10.825344 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.825272 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:54:10.895970 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.895938 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-registry-certificates\") pod \"00361e44-6e15-460e-88e7-1fd0398406e5\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " Oct 30 05:54:10.896133 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.895986 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-kube-api-access-fhp65\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-kube-api-access-fhp65\") pod \"00361e44-6e15-460e-88e7-1fd0398406e5\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " Oct 30 05:54:10.896133 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.896016 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\") pod \"00361e44-6e15-460e-88e7-1fd0398406e5\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " Oct 30 05:54:10.896133 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.896037 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-installation-pull-secrets\") pod \"00361e44-6e15-460e-88e7-1fd0398406e5\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " Oct 30 05:54:10.896287 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.896181 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-bound-sa-token\") pod \"00361e44-6e15-460e-88e7-1fd0398406e5\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " Oct 30 05:54:10.896287 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.896265 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00361e44-6e15-460e-88e7-1fd0398406e5-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00361e44-6e15-460e-88e7-1fd0398406e5-ca-trust-extracted\") pod \"00361e44-6e15-460e-88e7-1fd0398406e5\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " Oct 30 05:54:10.896426 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.896335 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-trusted-ca\") pod \"00361e44-6e15-460e-88e7-1fd0398406e5\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " Oct 30 05:54:10.896426 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.896385 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-image-registry-private-configuration\") pod \"00361e44-6e15-460e-88e7-1fd0398406e5\" (UID: \"00361e44-6e15-460e-88e7-1fd0398406e5\") " Oct 30 05:54:10.896541 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.896421 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-registry-certificates" pod "00361e44-6e15-460e-88e7-1fd0398406e5" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:10.896652 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.896630 5001 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-registry-certificates\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:54:10.896909 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.896851 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-trusted-ca" pod "00361e44-6e15-460e-88e7-1fd0398406e5" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:10.899720 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.899200 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-installation-pull-secrets" pod "00361e44-6e15-460e-88e7-1fd0398406e5" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:10.899720 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.899374 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls" pod "00361e44-6e15-460e-88e7-1fd0398406e5" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:10.899720 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.899614 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-kube-api-access-fhp65" pod "00361e44-6e15-460e-88e7-1fd0398406e5" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5"). InnerVolumeSpecName "kube-api-access-fhp65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:10.899979 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.899954 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-image-registry-private-configuration" pod "00361e44-6e15-460e-88e7-1fd0398406e5" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:10.900888 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.900849 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-bound-sa-token" pod "00361e44-6e15-460e-88e7-1fd0398406e5" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:10.904891 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.904861 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00361e44-6e15-460e-88e7-1fd0398406e5-ca-trust-extracted" pod "00361e44-6e15-460e-88e7-1fd0398406e5" (UID: "00361e44-6e15-460e-88e7-1fd0398406e5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:54:10.997190 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.997161 5001 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00361e44-6e15-460e-88e7-1fd0398406e5-ca-trust-extracted\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:54:10.997190 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.997188 5001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00361e44-6e15-460e-88e7-1fd0398406e5-trusted-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:54:10.997368 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.997199 5001 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-image-registry-private-configuration\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:54:10.997368 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.997208 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhp65\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-kube-api-access-fhp65\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:54:10.997368 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.997217 5001 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-registry-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:54:10.997368 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.997226 5001 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00361e44-6e15-460e-88e7-1fd0398406e5-installation-pull-secrets\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:54:10.997368 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:10.997234 5001 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00361e44-6e15-460e-88e7-1fd0398406e5-bound-sa-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 05:54:11.494600 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:11.494565 5001 generic.go:356] "Generic (PLEG): container finished" podID="00361e44-6e15-460e-88e7-1fd0398406e5" containerID="5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df" exitCode=0 Oct 30 05:54:11.494795 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:11.494606 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" event={"ID":"00361e44-6e15-460e-88e7-1fd0398406e5","Type":"ContainerDied","Data":"5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df"} Oct 30 05:54:11.494795 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:11.494631 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" Oct 30 05:54:11.494795 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:11.494666 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9c9cf4b4-89zvd" event={"ID":"00361e44-6e15-460e-88e7-1fd0398406e5","Type":"ContainerDied","Data":"e6b0d34a41c9abf63e904f2613bf2052bb8c638d0582cc4dac27cd3272480982"} Oct 30 05:54:11.494795 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:11.494686 5001 scope.go:122] "RemoveContainer" containerID="5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df" Oct 30 05:54:11.503598 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:11.503582 5001 scope.go:122] "RemoveContainer" containerID="5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df" Oct 30 05:54:11.503856 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:54:11.503839 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df\": container with ID starting with 5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df not found: ID does not exist" containerID="5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df" Oct 30 05:54:11.503893 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:11.503879 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df"} err="failed to get container status \"5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df\": rpc error: code = NotFound desc = could not find container \"5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df\": container with ID starting with 5e47801148b6c84f133ff324a0148da0f74b1b835dcba359615b3804010060df not found: ID does not exist" Oct 30 05:54:11.556662 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:11.556628 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b9c9cf4b4-89zvd"] Oct 30 05:54:11.559862 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:11.559846 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b9c9cf4b4-89zvd"] Oct 30 05:54:12.114670 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:12.114641 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00361e44-6e15-460e-88e7-1fd0398406e5" path="/var/lib/kubelet/pods/00361e44-6e15-460e-88e7-1fd0398406e5/volumes" Oct 30 05:54:21.539966 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:21.539932 5001 generic.go:356] "Generic (PLEG): container finished" podID="391996f2-6666-4d62-9af2-1291a66828bf" containerID="156b79b7901aa86644221a56d1ae524c2bd153ca8c64deb1d8ea0a6fd96ea9ea" exitCode=0 Oct 30 05:54:21.540424 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:21.540002 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" event={"ID":"391996f2-6666-4d62-9af2-1291a66828bf","Type":"ContainerDied","Data":"156b79b7901aa86644221a56d1ae524c2bd153ca8c64deb1d8ea0a6fd96ea9ea"} Oct 30 05:54:21.540424 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:21.540389 5001 scope.go:122] "RemoveContainer" containerID="156b79b7901aa86644221a56d1ae524c2bd153ca8c64deb1d8ea0a6fd96ea9ea" Oct 30 05:54:21.541669 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:21.541646 5001 generic.go:356] "Generic (PLEG): container finished" podID="67fb2af9-df3c-41a8-997c-e16de0871a3e" containerID="62e3999d9e3d7a38271da565e83facd860d9fd6ed78b657265d35e365556b854" exitCode=0 Oct 30 05:54:21.541739 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:21.541668 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" event={"ID":"67fb2af9-df3c-41a8-997c-e16de0871a3e","Type":"ContainerDied","Data":"62e3999d9e3d7a38271da565e83facd860d9fd6ed78b657265d35e365556b854"} Oct 30 05:54:21.541910 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:21.541890 5001 scope.go:122] "RemoveContainer" containerID="62e3999d9e3d7a38271da565e83facd860d9fd6ed78b657265d35e365556b854" Oct 30 05:54:22.547171 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:22.547133 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7chrf" event={"ID":"67fb2af9-df3c-41a8-997c-e16de0871a3e","Type":"ContainerStarted","Data":"cc3f13098d09fac2b055586a41ef1b4964ea77dd21bba15969bacf17f00367c2"} Oct 30 05:54:22.548824 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:54:22.548801 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6rbgq" event={"ID":"391996f2-6666-4d62-9af2-1291a66828bf","Type":"ContainerStarted","Data":"4aad2bbfb419f8db991d3f8b090cd26909e8cdc3f0414fdf6bf4f3b76c4759d0"} Oct 30 05:54:38.265098 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:54:38.265061 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:55:08.267716 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:55:08.267662 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:55:38.266004 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:55:38.265914 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:56:08.265775 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:56:08.265736 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:56:38.267367 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:56:38.267326 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:57:08.265062 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:57:08.265024 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:57:38.269592 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:57:38.269554 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:58:06.133596 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:58:06.133565 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 05:58:06.135125 ip-10-0-128-31 kubenswrapper[5001]: I1030 05:58:06.135106 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 05:58:08.265391 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:58:08.265355 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:58:38.265050 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:58:38.264936 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:59:08.264147 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:59:08.264106 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 05:59:38.266793 ip-10-0-128-31 kubenswrapper[5001]: E1030 05:59:38.266756 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:00:06.909602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909574 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v"] Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909848 5001 cpu_manager.go:414] "containerMap: removing container" podUID="00361e44-6e15-460e-88e7-1fd0398406e5" containerName="registry" Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909858 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00361e44-6e15-460e-88e7-1fd0398406e5" containerName="registry" Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909878 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e955d2b2-7e74-4bd9-a546-52c533182da7" containerName="insights-gathering" Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909883 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e955d2b2-7e74-4bd9-a546-52c533182da7" containerName="insights-gathering" Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909893 5001 cpu_manager.go:414] "containerMap: removing container" podUID="184d6270-19ed-4e6b-80c6-9734584b82a1" containerName="registry" Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909898 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184d6270-19ed-4e6b-80c6-9734584b82a1" containerName="registry" Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909909 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" containerName="insights-gathering" Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909914 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" containerName="insights-gathering" Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909961 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc92757-4a0b-44d6-9183-9d28c1b0d5e9" containerName="insights-gathering" Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909969 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="00361e44-6e15-460e-88e7-1fd0398406e5" containerName="registry" Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909978 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e955d2b2-7e74-4bd9-a546-52c533182da7" containerName="insights-gathering" Oct 30 06:00:06.909998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.909984 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="184d6270-19ed-4e6b-80c6-9734584b82a1" containerName="registry" Oct 30 06:00:06.913087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.913067 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:06.931897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.931866 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 06:00:06.932058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.931905 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 06:00:06.943717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.943695 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-vwjcj\"" Oct 30 06:00:06.964417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:06.964387 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v"] Oct 30 06:00:07.001621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.001584 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:07.001796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.001646 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7c8\" (UniqueName: \"kubernetes.io/projected/f142c145-ac67-474f-ba53-7621427335b0-kube-api-access-gs7c8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:07.001796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.001725 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:07.102251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.102210 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7c8\" (UniqueName: \"kubernetes.io/projected/f142c145-ac67-474f-ba53-7621427335b0-kube-api-access-gs7c8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:07.102469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.102266 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:07.102469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.102352 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:07.102767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.102747 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:07.102802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.102760 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:07.126928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.126906 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7c8\" (UniqueName: \"kubernetes.io/projected/f142c145-ac67-474f-ba53-7621427335b0-kube-api-access-gs7c8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:07.228217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.228118 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:07.549852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.549753 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v"] Oct 30 06:00:07.563942 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:07.563903 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf142c145_ac67_474f_ba53_7621427335b0.slice/crio-c9471d52b6909e79209454baaaaa87883ac2259e2bcf0fcf23217c8df85ce55c WatchSource:0}: Error finding container c9471d52b6909e79209454baaaaa87883ac2259e2bcf0fcf23217c8df85ce55c: Status 404 returned error can't find the container with id c9471d52b6909e79209454baaaaa87883ac2259e2bcf0fcf23217c8df85ce55c Oct 30 06:00:07.565787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.565768 5001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:00:07.760968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:07.760932 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" event={"ID":"f142c145-ac67-474f-ba53-7621427335b0","Type":"ContainerStarted","Data":"c9471d52b6909e79209454baaaaa87883ac2259e2bcf0fcf23217c8df85ce55c"} Oct 30 06:00:08.269921 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:00:08.269879 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:00:09.025037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.024994 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj"] Oct 30 06:00:09.029074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.029046 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:09.044568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.044537 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ctwm9\"" Oct 30 06:00:09.068602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.068567 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj"] Oct 30 06:00:09.118644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.118610 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:09.118855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.118679 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trjw\" (UniqueName: \"kubernetes.io/projected/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-kube-api-access-8trjw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:09.118855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.118837 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:09.127344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.127298 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr"] Oct 30 06:00:09.131101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.131079 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:09.146229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.146202 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-pr5fl\"" Oct 30 06:00:09.168389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.168354 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr"] Oct 30 06:00:09.217234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.217202 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj"] Oct 30 06:00:09.220201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.220172 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:09.220343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.220218 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:09.220343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.220249 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:09.220343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.220271 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qng92\" (UniqueName: \"kubernetes.io/projected/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-kube-api-access-qng92\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:09.220343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.220293 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:09.220595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.220376 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8trjw\" (UniqueName: \"kubernetes.io/projected/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-kube-api-access-8trjw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:09.220656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.220619 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:09.220656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.220645 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:09.220945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.220927 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:09.234953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.234930 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-b7njd\"" Oct 30 06:00:09.244151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.244123 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trjw\" (UniqueName: \"kubernetes.io/projected/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-kube-api-access-8trjw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:09.321384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.321272 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:09.321384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.321348 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qng92\" (UniqueName: \"kubernetes.io/projected/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-kube-api-access-qng92\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:09.321918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.321385 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:09.321918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.321433 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4grb\" (UniqueName: \"kubernetes.io/projected/326a4122-b3a3-43be-9423-aacf0d0f0a62-kube-api-access-j4grb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:09.321918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.321460 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:09.321918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.321507 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:09.321918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.321765 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:09.321918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.321866 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:09.322358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.322330 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn"] Oct 30 06:00:09.326475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.326454 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:09.328124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.328099 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj"] Oct 30 06:00:09.341060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.341035 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-95lgc\"" Oct 30 06:00:09.344785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.344762 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qng92\" (UniqueName: \"kubernetes.io/projected/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-kube-api-access-qng92\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:09.348506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.348485 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:09.422951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.422915 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:09.423113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.423018 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhv42\" (UniqueName: \"kubernetes.io/projected/e936521c-d0d1-4cc4-aa81-305cf22cd520-kube-api-access-rhv42\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:09.423113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.423057 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:09.423113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.423093 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:09.423295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.423141 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4grb\" (UniqueName: \"kubernetes.io/projected/326a4122-b3a3-43be-9423-aacf0d0f0a62-kube-api-access-j4grb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:09.423295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.423167 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:09.423664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.423641 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:09.423943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.423892 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:09.448278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.448242 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4grb\" (UniqueName: \"kubernetes.io/projected/326a4122-b3a3-43be-9423-aacf0d0f0a62-kube-api-access-j4grb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:09.449887 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.449861 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:09.470937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.470905 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn"] Oct 30 06:00:09.524208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.524174 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhv42\" (UniqueName: \"kubernetes.io/projected/e936521c-d0d1-4cc4-aa81-305cf22cd520-kube-api-access-rhv42\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:09.524389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.524232 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:09.524389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.524267 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:09.525352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.525276 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:09.525479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.525435 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:09.539109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.539088 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:09.549432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.549400 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhv42\" (UniqueName: \"kubernetes.io/projected/e936521c-d0d1-4cc4-aa81-305cf22cd520-kube-api-access-rhv42\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:09.645654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.645621 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:09.681334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.681286 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj"] Oct 30 06:00:09.722472 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:09.722434 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6551101a_0b79_44d7_bdd8_d0b50f78f6cd.slice/crio-8a8723f129287d5717b8d6e4c42b548653fd4262a8a7d753b9760610045cd38a WatchSource:0}: Error finding container 8a8723f129287d5717b8d6e4c42b548653fd4262a8a7d753b9760610045cd38a: Status 404 returned error can't find the container with id 8a8723f129287d5717b8d6e4c42b548653fd4262a8a7d753b9760610045cd38a Oct 30 06:00:09.773400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.773357 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" event={"ID":"6551101a-0b79-44d7-bdd8-d0b50f78f6cd","Type":"ContainerStarted","Data":"8a8723f129287d5717b8d6e4c42b548653fd4262a8a7d753b9760610045cd38a"} Oct 30 06:00:09.784103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.784069 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr"] Oct 30 06:00:09.797066 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:09.797030 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0fe77f_86f6_438a_92eb_f544c9d5c9c5.slice/crio-4e7fd056222488a1e7182752942bfa660d3be001b02963201640634295f6f120 WatchSource:0}: Error finding container 4e7fd056222488a1e7182752942bfa660d3be001b02963201640634295f6f120: Status 404 returned error can't find the container with id 4e7fd056222488a1e7182752942bfa660d3be001b02963201640634295f6f120 Oct 30 06:00:09.895814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.895773 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj"] Oct 30 06:00:09.936446 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:09.936417 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326a4122_b3a3_43be_9423_aacf0d0f0a62.slice/crio-5a27ed15d9ce44096ac6827357248fb55f2db6429fdf5531df2dffafe7cfa46d WatchSource:0}: Error finding container 5a27ed15d9ce44096ac6827357248fb55f2db6429fdf5531df2dffafe7cfa46d: Status 404 returned error can't find the container with id 5a27ed15d9ce44096ac6827357248fb55f2db6429fdf5531df2dffafe7cfa46d Oct 30 06:00:09.996523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:09.996488 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn"] Oct 30 06:00:10.039932 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:10.039892 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode936521c_d0d1_4cc4_aa81_305cf22cd520.slice/crio-a9350514f0c25df4709511dea73127de4f7daee2ad37067b38c87c297f31a26c WatchSource:0}: Error finding container a9350514f0c25df4709511dea73127de4f7daee2ad37067b38c87c297f31a26c: Status 404 returned error can't find the container with id a9350514f0c25df4709511dea73127de4f7daee2ad37067b38c87c297f31a26c Oct 30 06:00:10.534026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.532107 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv"] Oct 30 06:00:10.536826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.536118 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:10.552980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.552323 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-227tz\"" Oct 30 06:00:10.636574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.636516 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:10.636818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.636599 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p8sd\" (UniqueName: \"kubernetes.io/projected/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-kube-api-access-2p8sd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:10.636818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.636626 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:10.696413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.694404 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx"] Oct 30 06:00:10.699319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.698581 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:10.713918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.713627 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-rmcn5\"" Oct 30 06:00:10.739098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.737867 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p8sd\" (UniqueName: \"kubernetes.io/projected/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-kube-api-access-2p8sd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:10.739098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.737916 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:10.739098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.738011 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:10.739098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.738949 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:10.739983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.739937 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:10.763659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.763625 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p8sd\" (UniqueName: \"kubernetes.io/projected/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-kube-api-access-2p8sd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:10.770743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.770715 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv"] Oct 30 06:00:10.782862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.782827 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" event={"ID":"e936521c-d0d1-4cc4-aa81-305cf22cd520","Type":"ContainerStarted","Data":"a9350514f0c25df4709511dea73127de4f7daee2ad37067b38c87c297f31a26c"} Oct 30 06:00:10.784839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.784752 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" event={"ID":"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5","Type":"ContainerStarted","Data":"4e7fd056222488a1e7182752942bfa660d3be001b02963201640634295f6f120"} Oct 30 06:00:10.790339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.786812 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" event={"ID":"326a4122-b3a3-43be-9423-aacf0d0f0a62","Type":"ContainerStarted","Data":"5a27ed15d9ce44096ac6827357248fb55f2db6429fdf5531df2dffafe7cfa46d"} Oct 30 06:00:10.809705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.809671 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx"] Oct 30 06:00:10.839007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.838965 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:10.839007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.839009 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpl7f\" (UniqueName: \"kubernetes.io/projected/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-kube-api-access-qpl7f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:10.839205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.839152 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:10.858928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.858890 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:10.940034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.939975 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:10.940034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.940024 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpl7f\" (UniqueName: \"kubernetes.io/projected/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-kube-api-access-qpl7f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:10.940207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.940117 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:10.940575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.940554 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:10.940837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.940813 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:10.964662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:10.964633 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpl7f\" (UniqueName: \"kubernetes.io/projected/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-kube-api-access-qpl7f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:11.019732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:11.019696 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:11.257738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:11.257664 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv"] Oct 30 06:00:11.425792 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:11.425747 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd52f64_ce66_44c9_a7c5_5c719ddee7d2.slice/crio-43f97718a1da19b812daca63dd77a976d52af4c23d4c972659f47c66d0b33ba2 WatchSource:0}: Error finding container 43f97718a1da19b812daca63dd77a976d52af4c23d4c972659f47c66d0b33ba2: Status 404 returned error can't find the container with id 43f97718a1da19b812daca63dd77a976d52af4c23d4c972659f47c66d0b33ba2 Oct 30 06:00:11.507458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:11.507353 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx"] Oct 30 06:00:11.586376 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:11.586332 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3f9b10_60b0_4f8b_9563_fa3d1304d6ee.slice/crio-245de79aecb49799e987a9b1c7f3d2cd2fd6b3f97c373d3a9600dce4ca7654a4 WatchSource:0}: Error finding container 245de79aecb49799e987a9b1c7f3d2cd2fd6b3f97c373d3a9600dce4ca7654a4: Status 404 returned error can't find the container with id 245de79aecb49799e987a9b1c7f3d2cd2fd6b3f97c373d3a9600dce4ca7654a4 Oct 30 06:00:11.795034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:11.794945 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" event={"ID":"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee","Type":"ContainerStarted","Data":"245de79aecb49799e987a9b1c7f3d2cd2fd6b3f97c373d3a9600dce4ca7654a4"} Oct 30 06:00:11.796762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:11.796732 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" event={"ID":"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2","Type":"ContainerStarted","Data":"43f97718a1da19b812daca63dd77a976d52af4c23d4c972659f47c66d0b33ba2"} Oct 30 06:00:18.856672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:18.856629 5001 generic.go:356] "Generic (PLEG): container finished" podID="f142c145-ac67-474f-ba53-7621427335b0" containerID="0a8a181280540588f56886fc98b433195ac3aebe6dd7959b73090c30813604d6" exitCode=0 Oct 30 06:00:18.857167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:18.856736 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" event={"ID":"f142c145-ac67-474f-ba53-7621427335b0","Type":"ContainerDied","Data":"0a8a181280540588f56886fc98b433195ac3aebe6dd7959b73090c30813604d6"} Oct 30 06:00:18.863203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:18.863171 5001 generic.go:356] "Generic (PLEG): container finished" podID="326a4122-b3a3-43be-9423-aacf0d0f0a62" containerID="2c09176bb667c548390551a6781020161efd0e0d18d39277c1759a8564a1d338" exitCode=0 Oct 30 06:00:18.863359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:18.863294 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" event={"ID":"326a4122-b3a3-43be-9423-aacf0d0f0a62","Type":"ContainerDied","Data":"2c09176bb667c548390551a6781020161efd0e0d18d39277c1759a8564a1d338"} Oct 30 06:00:19.879463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:19.879410 5001 generic.go:356] "Generic (PLEG): container finished" podID="4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" containerID="497979ba34c7ee3dd009d3942b55a40a2e013c96d0f39209dc4ad26ff075e9b1" exitCode=0 Oct 30 06:00:19.879891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:19.879471 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" event={"ID":"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2","Type":"ContainerDied","Data":"497979ba34c7ee3dd009d3942b55a40a2e013c96d0f39209dc4ad26ff075e9b1"} Oct 30 06:00:20.887723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:20.886771 5001 generic.go:356] "Generic (PLEG): container finished" podID="f142c145-ac67-474f-ba53-7621427335b0" containerID="10add626a80b5f2c7bd539e678e50e4a81226dbe264d15433a313f1e49bb2ab2" exitCode=0 Oct 30 06:00:20.887723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:20.887451 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" event={"ID":"f142c145-ac67-474f-ba53-7621427335b0","Type":"ContainerDied","Data":"10add626a80b5f2c7bd539e678e50e4a81226dbe264d15433a313f1e49bb2ab2"} Oct 30 06:00:20.890931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:20.890350 5001 generic.go:356] "Generic (PLEG): container finished" podID="1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" containerID="5fce9d3cd2523bcca8997150901b662355f57b8087590b9a623eba2f5b21a91f" exitCode=0 Oct 30 06:00:20.890931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:20.890392 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" event={"ID":"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee","Type":"ContainerDied","Data":"5fce9d3cd2523bcca8997150901b662355f57b8087590b9a623eba2f5b21a91f"} Oct 30 06:00:21.897354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:21.897299 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" event={"ID":"e936521c-d0d1-4cc4-aa81-305cf22cd520","Type":"ContainerStarted","Data":"a8e5d8349e7c58792abb9c74f377285831c7cb34e3869abd19e69a7253c9fe7e"} Oct 30 06:00:21.898969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:21.898943 5001 generic.go:356] "Generic (PLEG): container finished" podID="326a4122-b3a3-43be-9423-aacf0d0f0a62" containerID="673f6505a78be17210689ebf615d5e2c3d056ee01458b94c88e92a6bb28406fb" exitCode=0 Oct 30 06:00:21.899068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:21.898999 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" event={"ID":"326a4122-b3a3-43be-9423-aacf0d0f0a62","Type":"ContainerDied","Data":"673f6505a78be17210689ebf615d5e2c3d056ee01458b94c88e92a6bb28406fb"} Oct 30 06:00:22.910082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:22.910044 5001 generic.go:356] "Generic (PLEG): container finished" podID="6551101a-0b79-44d7-bdd8-d0b50f78f6cd" containerID="180787c168c6031897614e81a737a40654bbd43b8b6ec11fc5077acc90b54e35" exitCode=0 Oct 30 06:00:22.910603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:22.910138 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" event={"ID":"6551101a-0b79-44d7-bdd8-d0b50f78f6cd","Type":"ContainerDied","Data":"180787c168c6031897614e81a737a40654bbd43b8b6ec11fc5077acc90b54e35"} Oct 30 06:00:22.913982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:22.913931 5001 generic.go:356] "Generic (PLEG): container finished" podID="e936521c-d0d1-4cc4-aa81-305cf22cd520" containerID="a8e5d8349e7c58792abb9c74f377285831c7cb34e3869abd19e69a7253c9fe7e" exitCode=0 Oct 30 06:00:22.914102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:22.914025 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" event={"ID":"e936521c-d0d1-4cc4-aa81-305cf22cd520","Type":"ContainerDied","Data":"a8e5d8349e7c58792abb9c74f377285831c7cb34e3869abd19e69a7253c9fe7e"} Oct 30 06:00:22.916998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:22.916966 5001 generic.go:356] "Generic (PLEG): container finished" podID="7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" containerID="a23214866e6022521c3a7e0c53cc737da976743ef5f026ef4ba3288f7423db6e" exitCode=0 Oct 30 06:00:22.917124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:22.917101 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" event={"ID":"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5","Type":"ContainerDied","Data":"a23214866e6022521c3a7e0c53cc737da976743ef5f026ef4ba3288f7423db6e"} Oct 30 06:00:23.839334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.838664 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g"] Oct 30 06:00:23.843247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.843227 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.859611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.859580 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 06:00:23.872982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.872942 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g"] Oct 30 06:00:23.873163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.873116 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hc86w\"" Oct 30 06:00:23.873939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.873908 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.874067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.873956 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.874067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.873990 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.874067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.874014 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.874067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.874042 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.874067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.874063 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.874338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.874103 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.874338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.874128 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.874338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.874169 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck7dk\" (UniqueName: \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-kube-api-access-ck7dk\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.974903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.974871 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.975216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.974935 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.975216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.974969 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.975216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.975009 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck7dk\" (UniqueName: \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-kube-api-access-ck7dk\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.975216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.975084 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.975216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.975120 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.975216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.975154 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.975216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.975179 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.975216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.975215 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.976591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.976541 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.976591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.976570 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.977065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.977041 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.977198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.977179 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.977342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.977324 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.982338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.981028 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:23.983028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:23.983001 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:24.014202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.014179 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:24.015755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.015720 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck7dk\" (UniqueName: \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-kube-api-access-ck7dk\") pod \"maas-default-gateway-openshift-default-58847f9fc5-l5d6g\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:24.061091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.060274 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7"] Oct 30 06:00:24.065667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.065030 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.076848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.076817 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a158a21-1de7-4e61-bfda-d255713a8655-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.077002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.076870 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a158a21-1de7-4e61-bfda-d255713a8655-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.077002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.076904 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.077002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.076950 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.077002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.076978 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a158a21-1de7-4e61-bfda-d255713a8655-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.077215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.077010 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.077215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.077039 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.077215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.077065 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jzvt\" (UniqueName: \"kubernetes.io/projected/7a158a21-1de7-4e61-bfda-d255713a8655-kube-api-access-8jzvt\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.077215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.077110 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.085826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.085766 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-xchzm\"" Oct 30 06:00:24.091810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.091789 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7"] Oct 30 06:00:24.163086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.163055 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:24.179425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.178776 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a158a21-1de7-4e61-bfda-d255713a8655-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.179425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.178828 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a158a21-1de7-4e61-bfda-d255713a8655-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.179425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.178861 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.179425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.178910 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.179425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.178939 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a158a21-1de7-4e61-bfda-d255713a8655-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.179425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.178973 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.179425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.179001 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.179425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.179040 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jzvt\" (UniqueName: \"kubernetes.io/projected/7a158a21-1de7-4e61-bfda-d255713a8655-kube-api-access-8jzvt\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.179425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.179105 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.180140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.180101 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a158a21-1de7-4e61-bfda-d255713a8655-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.180634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.180489 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.181696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.181394 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.181696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.181528 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.181696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.181608 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.184445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.184399 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a158a21-1de7-4e61-bfda-d255713a8655-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.188029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.187996 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a158a21-1de7-4e61-bfda-d255713a8655-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.204614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.204381 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a158a21-1de7-4e61-bfda-d255713a8655-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.205344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.205100 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jzvt\" (UniqueName: \"kubernetes.io/projected/7a158a21-1de7-4e61-bfda-d255713a8655-kube-api-access-8jzvt\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7\" (UID: \"7a158a21-1de7-4e61-bfda-d255713a8655\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.423417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.423390 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:24.516670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.515710 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g"] Oct 30 06:00:24.546145 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:24.546115 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c68d683_85ff_4411_bf67_d0f99a0c8d41.slice/crio-59304b27ceb74e9ba1c779f916cfd6c25aec9a49556c3d6f4a88b86055327d7f WatchSource:0}: Error finding container 59304b27ceb74e9ba1c779f916cfd6c25aec9a49556c3d6f4a88b86055327d7f: Status 404 returned error can't find the container with id 59304b27ceb74e9ba1c779f916cfd6c25aec9a49556c3d6f4a88b86055327d7f Oct 30 06:00:24.774886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.774843 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7"] Oct 30 06:00:24.794017 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:24.793983 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a158a21_1de7_4e61_bfda_d255713a8655.slice/crio-3c7f6f5d430d6f5617e0d965d8b81c1d56ba452188f0ae37ab6a6264659e5c7e WatchSource:0}: Error finding container 3c7f6f5d430d6f5617e0d965d8b81c1d56ba452188f0ae37ab6a6264659e5c7e: Status 404 returned error can't find the container with id 3c7f6f5d430d6f5617e0d965d8b81c1d56ba452188f0ae37ab6a6264659e5c7e Oct 30 06:00:24.931490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.931435 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" event={"ID":"7a158a21-1de7-4e61-bfda-d255713a8655","Type":"ContainerStarted","Data":"3c7f6f5d430d6f5617e0d965d8b81c1d56ba452188f0ae37ab6a6264659e5c7e"} Oct 30 06:00:24.933718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.933690 5001 generic.go:356] "Generic (PLEG): container finished" podID="6551101a-0b79-44d7-bdd8-d0b50f78f6cd" containerID="da97172968c6a337fa76c552c633aeccbca01ac82e90e46607e964fa03f5b08d" exitCode=0 Oct 30 06:00:24.933863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.933776 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" event={"ID":"6551101a-0b79-44d7-bdd8-d0b50f78f6cd","Type":"ContainerDied","Data":"da97172968c6a337fa76c552c633aeccbca01ac82e90e46607e964fa03f5b08d"} Oct 30 06:00:24.939958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.939288 5001 generic.go:356] "Generic (PLEG): container finished" podID="e936521c-d0d1-4cc4-aa81-305cf22cd520" containerID="e5e0dbb9d970fadb5acdc89af66e2edd45d5fc072863177b1e9759d876ccb7cf" exitCode=0 Oct 30 06:00:24.939958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.939411 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" event={"ID":"e936521c-d0d1-4cc4-aa81-305cf22cd520","Type":"ContainerDied","Data":"e5e0dbb9d970fadb5acdc89af66e2edd45d5fc072863177b1e9759d876ccb7cf"} Oct 30 06:00:24.941545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.941526 5001 generic.go:356] "Generic (PLEG): container finished" podID="7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" containerID="432ab0c5f0107a1a605d78cff1f05d5e587e8786dd10f0c6a3bb82be9a8a3c88" exitCode=0 Oct 30 06:00:24.941780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.941757 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" event={"ID":"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5","Type":"ContainerDied","Data":"432ab0c5f0107a1a605d78cff1f05d5e587e8786dd10f0c6a3bb82be9a8a3c88"} Oct 30 06:00:24.943330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.943285 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" event={"ID":"1c68d683-85ff-4411-bf67-d0f99a0c8d41","Type":"ContainerStarted","Data":"59304b27ceb74e9ba1c779f916cfd6c25aec9a49556c3d6f4a88b86055327d7f"} Oct 30 06:00:24.946509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.946471 5001 generic.go:356] "Generic (PLEG): container finished" podID="4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" containerID="4b04a3115063a18760e9ac13784de661d7f286446acda384bc3f1c7685fd01c1" exitCode=0 Oct 30 06:00:24.946609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:24.946517 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" event={"ID":"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2","Type":"ContainerDied","Data":"4b04a3115063a18760e9ac13784de661d7f286446acda384bc3f1c7685fd01c1"} Oct 30 06:00:31.450882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:31.450837 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:31.451215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:31.450930 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:31.451215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:31.450974 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:32.023355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:32.022837 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" event={"ID":"7a158a21-1de7-4e61-bfda-d255713a8655","Type":"ContainerStarted","Data":"ebfaa6038f56e771cab7929c61d2cb8b7b5df18de22c2ec96792d9e6e1ee61e6"} Oct 30 06:00:32.424335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:32.424220 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:32.430330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:32.430107 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:32.479199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:32.479138 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" podStartSLOduration=1.825956897 podStartE2EDuration="8.479118091s" totalImagesPullingTime="6.653161194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:32.091212398 +0000 UTC m=+446.087169444" watchObservedRunningTime="2026-10-30 06:00:32.479118091 +0000 UTC m=+446.475075103" Oct 30 06:00:33.029186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:33.029119 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:33.030682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:33.030454 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7" Oct 30 06:00:33.264350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:33.263587 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:33.264350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:33.263679 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:33.264350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:33.263727 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:34.041533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:34.041483 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" event={"ID":"1c68d683-85ff-4411-bf67-d0f99a0c8d41","Type":"ContainerStarted","Data":"b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b"} Oct 30 06:00:34.112135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:34.111488 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" podStartSLOduration=2.397585111 podStartE2EDuration="11.111466617s" totalImagesPullingTime="8.713881506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:34.097693568 +0000 UTC m=+448.093650579" watchObservedRunningTime="2026-10-30 06:00:34.111466617 +0000 UTC m=+448.107423628" Oct 30 06:00:34.166792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:34.164971 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:34.172060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:34.171810 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:35.048949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:35.048879 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:35.050965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:35.050762 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:00:36.060377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:36.060285 5001 generic.go:356] "Generic (PLEG): container finished" podID="326a4122-b3a3-43be-9423-aacf0d0f0a62" containerID="733e7df546b89546a2005b52e41988be3ef26e95eae72b41c5045cee963f4bd9" exitCode=0 Oct 30 06:00:36.060792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:36.060485 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" event={"ID":"326a4122-b3a3-43be-9423-aacf0d0f0a62","Type":"ContainerDied","Data":"733e7df546b89546a2005b52e41988be3ef26e95eae72b41c5045cee963f4bd9"} Oct 30 06:00:36.064659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:36.064622 5001 generic.go:356] "Generic (PLEG): container finished" podID="f142c145-ac67-474f-ba53-7621427335b0" containerID="03567a6e98b2eb90df062a3b48039495c0ee449aa683800408c5273f19617c9b" exitCode=0 Oct 30 06:00:36.065468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:36.065438 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" event={"ID":"f142c145-ac67-474f-ba53-7621427335b0","Type":"ContainerDied","Data":"03567a6e98b2eb90df062a3b48039495c0ee449aa683800408c5273f19617c9b"} Oct 30 06:00:37.076142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.076102 5001 generic.go:356] "Generic (PLEG): container finished" podID="1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" containerID="60eb521668d1bab9fa7684df88331d457c4327b7b88da3360617c0f75c1ad4ba" exitCode=0 Oct 30 06:00:37.077173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.077141 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" event={"ID":"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee","Type":"ContainerDied","Data":"60eb521668d1bab9fa7684df88331d457c4327b7b88da3360617c0f75c1ad4ba"} Oct 30 06:00:37.254784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.254537 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:37.273083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.272159 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:37.321190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.314638 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-bundle\" (UniqueName: \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-bundle\") pod \"326a4122-b3a3-43be-9423-aacf0d0f0a62\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " Oct 30 06:00:37.321190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.314765 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326a4122-b3a3-43be-9423-aacf0d0f0a62-kube-api-access-j4grb\" (UniqueName: \"kubernetes.io/projected/326a4122-b3a3-43be-9423-aacf0d0f0a62-kube-api-access-j4grb\") pod \"326a4122-b3a3-43be-9423-aacf0d0f0a62\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " Oct 30 06:00:37.321190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.314791 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-util\" (UniqueName: \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-util\") pod \"f142c145-ac67-474f-ba53-7621427335b0\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " Oct 30 06:00:37.321190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.314823 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-util\" (UniqueName: \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-util\") pod \"326a4122-b3a3-43be-9423-aacf0d0f0a62\" (UID: \"326a4122-b3a3-43be-9423-aacf0d0f0a62\") " Oct 30 06:00:37.321190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.314862 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f142c145-ac67-474f-ba53-7621427335b0-kube-api-access-gs7c8\" (UniqueName: \"kubernetes.io/projected/f142c145-ac67-474f-ba53-7621427335b0-kube-api-access-gs7c8\") pod \"f142c145-ac67-474f-ba53-7621427335b0\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " Oct 30 06:00:37.321190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.314916 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-bundle\") pod \"f142c145-ac67-474f-ba53-7621427335b0\" (UID: \"f142c145-ac67-474f-ba53-7621427335b0\") " Oct 30 06:00:37.321190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.316081 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-bundle" pod "f142c145-ac67-474f-ba53-7621427335b0" (UID: "f142c145-ac67-474f-ba53-7621427335b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:37.321190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.316158 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-bundle" pod "326a4122-b3a3-43be-9423-aacf0d0f0a62" (UID: "326a4122-b3a3-43be-9423-aacf0d0f0a62"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:37.321190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.318593 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326a4122-b3a3-43be-9423-aacf0d0f0a62-kube-api-access-j4grb" pod "326a4122-b3a3-43be-9423-aacf0d0f0a62" (UID: "326a4122-b3a3-43be-9423-aacf0d0f0a62"). InnerVolumeSpecName "kube-api-access-j4grb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:37.321190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.318922 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f142c145-ac67-474f-ba53-7621427335b0-kube-api-access-gs7c8" pod "f142c145-ac67-474f-ba53-7621427335b0" (UID: "f142c145-ac67-474f-ba53-7621427335b0"). InnerVolumeSpecName "kube-api-access-gs7c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:37.325550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.325516 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-util" pod "326a4122-b3a3-43be-9423-aacf0d0f0a62" (UID: "326a4122-b3a3-43be-9423-aacf0d0f0a62"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:37.327332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.327248 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-util" pod "f142c145-ac67-474f-ba53-7621427335b0" (UID: "f142c145-ac67-474f-ba53-7621427335b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:37.415913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.415867 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:37.415913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.415915 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:37.416174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.415931 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4grb\" (UniqueName: \"kubernetes.io/projected/326a4122-b3a3-43be-9423-aacf0d0f0a62-kube-api-access-j4grb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:37.416174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.415945 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f142c145-ac67-474f-ba53-7621427335b0-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:37.416174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.415958 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/326a4122-b3a3-43be-9423-aacf0d0f0a62-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:37.416174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:37.415971 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs7c8\" (UniqueName: \"kubernetes.io/projected/f142c145-ac67-474f-ba53-7621427335b0-kube-api-access-gs7c8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:38.087719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:38.086280 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" event={"ID":"f142c145-ac67-474f-ba53-7621427335b0","Type":"ContainerDied","Data":"c9471d52b6909e79209454baaaaa87883ac2259e2bcf0fcf23217c8df85ce55c"} Oct 30 06:00:38.087719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:38.086403 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9471d52b6909e79209454baaaaa87883ac2259e2bcf0fcf23217c8df85ce55c" Oct 30 06:00:38.087719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:38.086477 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712x89v" Oct 30 06:00:38.091297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:38.091036 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" event={"ID":"326a4122-b3a3-43be-9423-aacf0d0f0a62","Type":"ContainerDied","Data":"5a27ed15d9ce44096ac6827357248fb55f2db6429fdf5531df2dffafe7cfa46d"} Oct 30 06:00:38.091297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:38.091088 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a27ed15d9ce44096ac6827357248fb55f2db6429fdf5531df2dffafe7cfa46d" Oct 30 06:00:38.091297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:38.091173 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhzrj" Oct 30 06:00:38.098082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:38.098038 5001 generic.go:356] "Generic (PLEG): container finished" podID="1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" containerID="6fb0d37f20aa0248e55b2eac843ec187c9e75a946802120586df7deffc2613cd" exitCode=0 Oct 30 06:00:38.098225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:38.098136 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" event={"ID":"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee","Type":"ContainerDied","Data":"6fb0d37f20aa0248e55b2eac843ec187c9e75a946802120586df7deffc2613cd"} Oct 30 06:00:38.282012 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:00:38.281970 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:00:39.295816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:39.295781 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:39.332605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:39.332573 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-kube-api-access-qpl7f\" (UniqueName: \"kubernetes.io/projected/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-kube-api-access-qpl7f\") pod \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " Oct 30 06:00:39.332784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:39.332632 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-bundle\") pod \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " Oct 30 06:00:39.332784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:39.332693 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-util\" (UniqueName: \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-util\") pod \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\" (UID: \"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee\") " Oct 30 06:00:39.333065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:39.333031 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-bundle" pod "1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" (UID: "1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:39.334817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:39.334797 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-kube-api-access-qpl7f" pod "1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" (UID: "1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee"). InnerVolumeSpecName "kube-api-access-qpl7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:39.338190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:39.338156 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-util" pod "1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" (UID: "1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:39.433401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:39.433359 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:39.433401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:39.433406 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:39.433585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:39.433416 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpl7f\" (UniqueName: \"kubernetes.io/projected/1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee-kube-api-access-qpl7f\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:40.110785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:40.110750 5001 generic.go:356] "Generic (PLEG): container finished" podID="6551101a-0b79-44d7-bdd8-d0b50f78f6cd" containerID="96f4c3527ecbef10de099020a685528ac1c73dc9a846032f2c4884bcace46a5c" exitCode=0 Oct 30 06:00:40.113067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:40.113040 5001 generic.go:356] "Generic (PLEG): container finished" podID="e936521c-d0d1-4cc4-aa81-305cf22cd520" containerID="f3ccc350ee702f6fcae05b1128886e68cceefcd0f3921bb6cc2871341d0c9220" exitCode=0 Oct 30 06:00:40.114490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:40.114459 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" event={"ID":"6551101a-0b79-44d7-bdd8-d0b50f78f6cd","Type":"ContainerDied","Data":"96f4c3527ecbef10de099020a685528ac1c73dc9a846032f2c4884bcace46a5c"} Oct 30 06:00:40.114635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:40.114498 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" event={"ID":"e936521c-d0d1-4cc4-aa81-305cf22cd520","Type":"ContainerDied","Data":"f3ccc350ee702f6fcae05b1128886e68cceefcd0f3921bb6cc2871341d0c9220"} Oct 30 06:00:40.115185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:40.115162 5001 generic.go:356] "Generic (PLEG): container finished" podID="7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" containerID="dd940e56aa821a8a0f86ded15c1573bb049b135e611a3b1ef4c2a03cad980740" exitCode=0 Oct 30 06:00:40.115279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:40.115257 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" event={"ID":"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5","Type":"ContainerDied","Data":"dd940e56aa821a8a0f86ded15c1573bb049b135e611a3b1ef4c2a03cad980740"} Oct 30 06:00:40.117105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:40.117083 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" event={"ID":"1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee","Type":"ContainerDied","Data":"245de79aecb49799e987a9b1c7f3d2cd2fd6b3f97c373d3a9600dce4ca7654a4"} Oct 30 06:00:40.117105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:40.117097 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gfmpx" Oct 30 06:00:40.117252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:40.117120 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="245de79aecb49799e987a9b1c7f3d2cd2fd6b3f97c373d3a9600dce4ca7654a4" Oct 30 06:00:40.119073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:40.119049 5001 generic.go:356] "Generic (PLEG): container finished" podID="4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" containerID="42c58d065b596dc4ec9d32798db684d7afade0b723f66932fe4024fc267daa4f" exitCode=0 Oct 30 06:00:40.119168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:40.119085 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" event={"ID":"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2","Type":"ContainerDied","Data":"42c58d065b596dc4ec9d32798db684d7afade0b723f66932fe4024fc267daa4f"} Oct 30 06:00:41.057769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.057739 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4"] Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058664 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f142c145-ac67-474f-ba53-7621427335b0" containerName="util" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058716 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f142c145-ac67-474f-ba53-7621427335b0" containerName="util" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058745 5001 cpu_manager.go:414] "containerMap: removing container" podUID="326a4122-b3a3-43be-9423-aacf0d0f0a62" containerName="extract" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058754 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326a4122-b3a3-43be-9423-aacf0d0f0a62" containerName="extract" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058776 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f142c145-ac67-474f-ba53-7621427335b0" containerName="pull" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058784 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f142c145-ac67-474f-ba53-7621427335b0" containerName="pull" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058813 5001 cpu_manager.go:414] "containerMap: removing container" podUID="326a4122-b3a3-43be-9423-aacf0d0f0a62" containerName="util" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058820 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326a4122-b3a3-43be-9423-aacf0d0f0a62" containerName="util" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058833 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" containerName="pull" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058840 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" containerName="pull" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058864 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f142c145-ac67-474f-ba53-7621427335b0" containerName="extract" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058871 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f142c145-ac67-474f-ba53-7621427335b0" containerName="extract" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058911 5001 cpu_manager.go:414] "containerMap: removing container" podUID="326a4122-b3a3-43be-9423-aacf0d0f0a62" containerName="pull" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058919 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326a4122-b3a3-43be-9423-aacf0d0f0a62" containerName="pull" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058935 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" containerName="extract" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058941 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" containerName="extract" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058961 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" containerName="util" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.058968 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" containerName="util" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.059210 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="326a4122-b3a3-43be-9423-aacf0d0f0a62" containerName="extract" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.059239 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f142c145-ac67-474f-ba53-7621427335b0" containerName="extract" Oct 30 06:00:41.061917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.059253 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3f9b10-60b0-4f8b-9563-fa3d1304d6ee" containerName="extract" Oct 30 06:00:41.064563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.064534 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.083707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.083686 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 06:00:41.083837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.083704 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 06:00:41.083837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.083751 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 06:00:41.094801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.094783 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n5fhg\"" Oct 30 06:00:41.094901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.094815 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 06:00:41.149869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.149828 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93d09621-eaf7-4eac-8dfd-dce91b37e395-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.150041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.149931 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9vmw\" (UniqueName: \"kubernetes.io/projected/93d09621-eaf7-4eac-8dfd-dce91b37e395-kube-api-access-h9vmw\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.150041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.150019 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93d09621-eaf7-4eac-8dfd-dce91b37e395-tmp\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.150127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.150081 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93d09621-eaf7-4eac-8dfd-dce91b37e395-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.150127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.150116 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/93d09621-eaf7-4eac-8dfd-dce91b37e395-manifests\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.150200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.150159 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/93d09621-eaf7-4eac-8dfd-dce91b37e395-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.183744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.183703 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 06:00:41.250926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.250893 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93d09621-eaf7-4eac-8dfd-dce91b37e395-tmp\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.251074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.250961 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93d09621-eaf7-4eac-8dfd-dce91b37e395-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.251074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.251003 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/93d09621-eaf7-4eac-8dfd-dce91b37e395-manifests\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.251074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.251042 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/93d09621-eaf7-4eac-8dfd-dce91b37e395-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.251074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.251066 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93d09621-eaf7-4eac-8dfd-dce91b37e395-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.251290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.251108 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9vmw\" (UniqueName: \"kubernetes.io/projected/93d09621-eaf7-4eac-8dfd-dce91b37e395-kube-api-access-h9vmw\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.251755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.251729 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/93d09621-eaf7-4eac-8dfd-dce91b37e395-manifests\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.254656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.254632 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93d09621-eaf7-4eac-8dfd-dce91b37e395-tmp\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.255069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.255047 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93d09621-eaf7-4eac-8dfd-dce91b37e395-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.255171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.255152 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/93d09621-eaf7-4eac-8dfd-dce91b37e395-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.256095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.256039 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93d09621-eaf7-4eac-8dfd-dce91b37e395-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.263980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.263957 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4"] Oct 30 06:00:41.283035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.283004 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9vmw\" (UniqueName: \"kubernetes.io/projected/93d09621-eaf7-4eac-8dfd-dce91b37e395-kube-api-access-h9vmw\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-xzzq4\" (UID: \"93d09621-eaf7-4eac-8dfd-dce91b37e395\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.312065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.312001 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:41.314110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.314089 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:41.334470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.334441 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:41.341049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.341025 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:41.351526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.351506 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-util\" (UniqueName: \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-util\") pod \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " Oct 30 06:00:41.351588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.351578 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-bundle\") pod \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " Oct 30 06:00:41.351628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.351606 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-bundle\") pod \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " Oct 30 06:00:41.351660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.351637 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-kube-api-access-8trjw\" (UniqueName: \"kubernetes.io/projected/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-kube-api-access-8trjw\") pod \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\" (UID: \"6551101a-0b79-44d7-bdd8-d0b50f78f6cd\") " Oct 30 06:00:41.351693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.351678 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-kube-api-access-qng92\" (UniqueName: \"kubernetes.io/projected/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-kube-api-access-qng92\") pod \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " Oct 30 06:00:41.351746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.351724 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-util\" (UniqueName: \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-util\") pod \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\" (UID: \"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5\") " Oct 30 06:00:41.352159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.352131 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-bundle" pod "7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" (UID: "7d0fe77f-86f6-438a-92eb-f544c9d5c9c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:41.352393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.352368 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-bundle" pod "6551101a-0b79-44d7-bdd8-d0b50f78f6cd" (UID: "6551101a-0b79-44d7-bdd8-d0b50f78f6cd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:41.353871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.353851 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-kube-api-access-8trjw" pod "6551101a-0b79-44d7-bdd8-d0b50f78f6cd" (UID: "6551101a-0b79-44d7-bdd8-d0b50f78f6cd"). InnerVolumeSpecName "kube-api-access-8trjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:41.354181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.354158 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-kube-api-access-qng92" pod "7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" (UID: "7d0fe77f-86f6-438a-92eb-f544c9d5c9c5"). InnerVolumeSpecName "kube-api-access-qng92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:41.356969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.356885 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-util" pod "6551101a-0b79-44d7-bdd8-d0b50f78f6cd" (UID: "6551101a-0b79-44d7-bdd8-d0b50f78f6cd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:41.357569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.357539 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-util" pod "7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" (UID: "7d0fe77f-86f6-438a-92eb-f544c9d5c9c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:41.380029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.380005 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:41.452426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452396 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-bundle\") pod \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " Oct 30 06:00:41.452722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452472 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-bundle\") pod \"e936521c-d0d1-4cc4-aa81-305cf22cd520\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " Oct 30 06:00:41.452722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452523 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e936521c-d0d1-4cc4-aa81-305cf22cd520-kube-api-access-rhv42\" (UniqueName: \"kubernetes.io/projected/e936521c-d0d1-4cc4-aa81-305cf22cd520-kube-api-access-rhv42\") pod \"e936521c-d0d1-4cc4-aa81-305cf22cd520\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " Oct 30 06:00:41.452722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452576 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-kube-api-access-2p8sd\" (UniqueName: \"kubernetes.io/projected/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-kube-api-access-2p8sd\") pod \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " Oct 30 06:00:41.452722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452606 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-util\" (UniqueName: \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-util\") pod \"e936521c-d0d1-4cc4-aa81-305cf22cd520\" (UID: \"e936521c-d0d1-4cc4-aa81-305cf22cd520\") " Oct 30 06:00:41.452722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452629 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-util\" (UniqueName: \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-util\") pod \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\" (UID: \"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2\") " Oct 30 06:00:41.454815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452892 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.454815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452908 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.454815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452919 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.454815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452927 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8trjw\" (UniqueName: \"kubernetes.io/projected/6551101a-0b79-44d7-bdd8-d0b50f78f6cd-kube-api-access-8trjw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.454815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452937 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qng92\" (UniqueName: \"kubernetes.io/projected/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-kube-api-access-qng92\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.454815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.452945 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d0fe77f-86f6-438a-92eb-f544c9d5c9c5-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.454815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.454071 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-bundle" pod "e936521c-d0d1-4cc4-aa81-305cf22cd520" (UID: "e936521c-d0d1-4cc4-aa81-305cf22cd520"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:41.457277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.454910 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-bundle" pod "4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" (UID: "4cd52f64-ce66-44c9-a7c5-5c719ddee7d2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:41.457277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.456668 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e936521c-d0d1-4cc4-aa81-305cf22cd520-kube-api-access-rhv42" pod "e936521c-d0d1-4cc4-aa81-305cf22cd520" (UID: "e936521c-d0d1-4cc4-aa81-305cf22cd520"). InnerVolumeSpecName "kube-api-access-rhv42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:41.458590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.458555 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-kube-api-access-2p8sd" pod "4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" (UID: "4cd52f64-ce66-44c9-a7c5-5c719ddee7d2"). InnerVolumeSpecName "kube-api-access-2p8sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:41.459889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.459851 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-util" pod "4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" (UID: "4cd52f64-ce66-44c9-a7c5-5c719ddee7d2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:41.462203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.462174 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-util" pod "e936521c-d0d1-4cc4-aa81-305cf22cd520" (UID: "e936521c-d0d1-4cc4-aa81-305cf22cd520"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:41.553723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.553688 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhv42\" (UniqueName: \"kubernetes.io/projected/e936521c-d0d1-4cc4-aa81-305cf22cd520-kube-api-access-rhv42\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.553723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.553719 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p8sd\" (UniqueName: \"kubernetes.io/projected/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-kube-api-access-2p8sd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.553723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.553729 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.553960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.553739 5001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.553960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.553747 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cd52f64-ce66-44c9-a7c5-5c719ddee7d2-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.553960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.553755 5001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e936521c-d0d1-4cc4-aa81-305cf22cd520-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:00:41.780632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:41.780598 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4"] Oct 30 06:00:41.810797 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:41.810771 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d09621_eaf7_4eac_8dfd_dce91b37e395.slice/crio-5a329cb46255996d19cb796768ee4cbca97445831bc316761f74444cdf1febb8 WatchSource:0}: Error finding container 5a329cb46255996d19cb796768ee4cbca97445831bc316761f74444cdf1febb8: Status 404 returned error can't find the container with id 5a329cb46255996d19cb796768ee4cbca97445831bc316761f74444cdf1febb8 Oct 30 06:00:42.128578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.128537 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" event={"ID":"93d09621-eaf7-4eac-8dfd-dce91b37e395","Type":"ContainerStarted","Data":"5a329cb46255996d19cb796768ee4cbca97445831bc316761f74444cdf1febb8"} Oct 30 06:00:42.130585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.130559 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" event={"ID":"6551101a-0b79-44d7-bdd8-d0b50f78f6cd","Type":"ContainerDied","Data":"8a8723f129287d5717b8d6e4c42b548653fd4262a8a7d753b9760610045cd38a"} Oct 30 06:00:42.130681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.130599 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a8723f129287d5717b8d6e4c42b548653fd4262a8a7d753b9760610045cd38a" Oct 30 06:00:42.130681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.130569 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advfmj" Oct 30 06:00:42.132744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.132717 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" event={"ID":"e936521c-d0d1-4cc4-aa81-305cf22cd520","Type":"ContainerDied","Data":"a9350514f0c25df4709511dea73127de4f7daee2ad37067b38c87c297f31a26c"} Oct 30 06:00:42.132881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.132759 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zddn" Oct 30 06:00:42.132881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.132766 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9350514f0c25df4709511dea73127de4f7daee2ad37067b38c87c297f31a26c" Oct 30 06:00:42.134725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.134702 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" Oct 30 06:00:42.134810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.134744 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c644rqr" event={"ID":"7d0fe77f-86f6-438a-92eb-f544c9d5c9c5","Type":"ContainerDied","Data":"4e7fd056222488a1e7182752942bfa660d3be001b02963201640634295f6f120"} Oct 30 06:00:42.134810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.134800 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e7fd056222488a1e7182752942bfa660d3be001b02963201640634295f6f120" Oct 30 06:00:42.136780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.136760 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" event={"ID":"4cd52f64-ce66-44c9-a7c5-5c719ddee7d2","Type":"ContainerDied","Data":"43f97718a1da19b812daca63dd77a976d52af4c23d4c972659f47c66d0b33ba2"} Oct 30 06:00:42.136880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.136789 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43f97718a1da19b812daca63dd77a976d52af4c23d4c972659f47c66d0b33ba2" Oct 30 06:00:42.136880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:42.136809 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvksmv" Oct 30 06:00:45.156976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:45.156944 5001 generic.go:356] "Generic (PLEG): container finished" podID="93d09621-eaf7-4eac-8dfd-dce91b37e395" containerID="868905df09bbd617dae1604dab61a9f54a13da50108f52ee11b7d7ea7badfd6b" exitCode=0 Oct 30 06:00:45.157419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:45.157019 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" event={"ID":"93d09621-eaf7-4eac-8dfd-dce91b37e395","Type":"ContainerDied","Data":"868905df09bbd617dae1604dab61a9f54a13da50108f52ee11b7d7ea7badfd6b"} Oct 30 06:00:46.162914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:46.162881 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" event={"ID":"93d09621-eaf7-4eac-8dfd-dce91b37e395","Type":"ContainerStarted","Data":"6b241a7eba833307ceb52756f28bf175cac3f7dabd5635ee2d4dce07885842a9"} Oct 30 06:00:46.163326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:46.162998 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:46.271198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:46.271142 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" podStartSLOduration=2.6003544979999997 podStartE2EDuration="6.271127074s" totalImagesPullingTime="3.252987972s" totalInitContainerRuntime="417.784604ms" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:46.258122307 +0000 UTC m=+460.254079317" watchObservedRunningTime="2026-10-30 06:00:46.271127074 +0000 UTC m=+460.267084084" Oct 30 06:00:51.970193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970158 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt"] Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970482 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" containerName="util" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970493 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" containerName="util" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970502 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" containerName="pull" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970507 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" containerName="pull" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970517 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e936521c-d0d1-4cc4-aa81-305cf22cd520" containerName="util" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970523 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e936521c-d0d1-4cc4-aa81-305cf22cd520" containerName="util" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970531 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6551101a-0b79-44d7-bdd8-d0b50f78f6cd" containerName="util" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970535 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6551101a-0b79-44d7-bdd8-d0b50f78f6cd" containerName="util" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970542 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6551101a-0b79-44d7-bdd8-d0b50f78f6cd" containerName="extract" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970545 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6551101a-0b79-44d7-bdd8-d0b50f78f6cd" containerName="extract" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970556 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6551101a-0b79-44d7-bdd8-d0b50f78f6cd" containerName="pull" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970560 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6551101a-0b79-44d7-bdd8-d0b50f78f6cd" containerName="pull" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970565 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e936521c-d0d1-4cc4-aa81-305cf22cd520" containerName="pull" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970569 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e936521c-d0d1-4cc4-aa81-305cf22cd520" containerName="pull" Oct 30 06:00:51.970573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970581 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" containerName="extract" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970585 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" containerName="extract" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970592 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" containerName="pull" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970597 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" containerName="pull" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970606 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" containerName="extract" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970610 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" containerName="extract" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970616 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" containerName="util" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970620 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" containerName="util" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970628 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e936521c-d0d1-4cc4-aa81-305cf22cd520" containerName="extract" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970632 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e936521c-d0d1-4cc4-aa81-305cf22cd520" containerName="extract" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970679 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0fe77f-86f6-438a-92eb-f544c9d5c9c5" containerName="extract" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970687 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e936521c-d0d1-4cc4-aa81-305cf22cd520" containerName="extract" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970695 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6551101a-0b79-44d7-bdd8-d0b50f78f6cd" containerName="extract" Oct 30 06:00:51.971008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.970702 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd52f64-ce66-44c9-a7c5-5c719ddee7d2" containerName="extract" Oct 30 06:00:51.974549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.974526 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt" Oct 30 06:00:51.992778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:51.992753 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt"] Oct 30 06:00:52.002214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:52.002186 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 06:00:52.002214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:52.002188 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:00:52.002456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:52.002252 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qzvpr\"" Oct 30 06:00:52.044750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:52.044715 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/982bc3ce-f3a0-4030-a534-9e36f5d834ff-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt\" (UID: \"982bc3ce-f3a0-4030-a534-9e36f5d834ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt" Oct 30 06:00:52.044935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:52.044758 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm8bg\" (UniqueName: \"kubernetes.io/projected/982bc3ce-f3a0-4030-a534-9e36f5d834ff-kube-api-access-qm8bg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt\" (UID: \"982bc3ce-f3a0-4030-a534-9e36f5d834ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt" Oct 30 06:00:52.145781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:52.145750 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/982bc3ce-f3a0-4030-a534-9e36f5d834ff-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt\" (UID: \"982bc3ce-f3a0-4030-a534-9e36f5d834ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt" Oct 30 06:00:52.145950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:52.145786 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm8bg\" (UniqueName: \"kubernetes.io/projected/982bc3ce-f3a0-4030-a534-9e36f5d834ff-kube-api-access-qm8bg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt\" (UID: \"982bc3ce-f3a0-4030-a534-9e36f5d834ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt" Oct 30 06:00:52.146116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:52.146095 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/982bc3ce-f3a0-4030-a534-9e36f5d834ff-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt\" (UID: \"982bc3ce-f3a0-4030-a534-9e36f5d834ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt" Oct 30 06:00:52.169949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:52.169921 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm8bg\" (UniqueName: \"kubernetes.io/projected/982bc3ce-f3a0-4030-a534-9e36f5d834ff-kube-api-access-qm8bg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt\" (UID: \"982bc3ce-f3a0-4030-a534-9e36f5d834ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt" Oct 30 06:00:52.285215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:52.285128 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt" Oct 30 06:00:52.616991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:52.616955 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt"] Oct 30 06:00:52.621801 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:52.621774 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982bc3ce_f3a0_4030_a534_9e36f5d834ff.slice/crio-58118ed6fe1755d1388ad5fa4bb2b925ffd21f34960775e5040d27ccc67f95b5 WatchSource:0}: Error finding container 58118ed6fe1755d1388ad5fa4bb2b925ffd21f34960775e5040d27ccc67f95b5: Status 404 returned error can't find the container with id 58118ed6fe1755d1388ad5fa4bb2b925ffd21f34960775e5040d27ccc67f95b5 Oct 30 06:00:53.196464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:53.196428 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt" event={"ID":"982bc3ce-f3a0-4030-a534-9e36f5d834ff","Type":"ContainerStarted","Data":"58118ed6fe1755d1388ad5fa4bb2b925ffd21f34960775e5040d27ccc67f95b5"} Oct 30 06:00:55.208784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:55.208753 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt" event={"ID":"982bc3ce-f3a0-4030-a534-9e36f5d834ff","Type":"ContainerStarted","Data":"83cbd9d66f7067809d61d9495282b5b30e31a890f639c30be477a52eb09e84b7"} Oct 30 06:00:55.286940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:55.286888 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rzxrt" podStartSLOduration=2.716639421 podStartE2EDuration="4.286872909s" totalImagesPullingTime="1.570233488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:55.285737908 +0000 UTC m=+469.281695131" watchObservedRunningTime="2026-10-30 06:00:55.286872909 +0000 UTC m=+469.282829919" Oct 30 06:00:57.170241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:57.170212 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-xzzq4" Oct 30 06:00:58.595124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:58.595086 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg"] Oct 30 06:00:58.597544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:58.597529 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg" Oct 30 06:00:58.634271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:58.634244 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zfl9m\"" Oct 30 06:00:58.634421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:58.634288 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 06:00:58.635925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:58.635908 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 06:00:58.640667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:58.640640 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg"] Oct 30 06:00:58.707274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:58.707238 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnjhr\" (UniqueName: \"kubernetes.io/projected/8f6114c0-230e-40ef-9cb5-60f49d0f4825-kube-api-access-bnjhr\") pod \"limitador-operator-controller-manager-5778bb4449-thdmg\" (UID: \"8f6114c0-230e-40ef-9cb5-60f49d0f4825\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg" Oct 30 06:00:58.808406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:58.808370 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjhr\" (UniqueName: \"kubernetes.io/projected/8f6114c0-230e-40ef-9cb5-60f49d0f4825-kube-api-access-bnjhr\") pod \"limitador-operator-controller-manager-5778bb4449-thdmg\" (UID: \"8f6114c0-230e-40ef-9cb5-60f49d0f4825\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg" Oct 30 06:00:58.861140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:58.861114 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjhr\" (UniqueName: \"kubernetes.io/projected/8f6114c0-230e-40ef-9cb5-60f49d0f4825-kube-api-access-bnjhr\") pod \"limitador-operator-controller-manager-5778bb4449-thdmg\" (UID: \"8f6114c0-230e-40ef-9cb5-60f49d0f4825\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg" Oct 30 06:00:58.908754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:58.908716 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg" Oct 30 06:00:59.228339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:00:59.228241 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg"] Oct 30 06:00:59.232063 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:00:59.232037 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6114c0_230e_40ef_9cb5_60f49d0f4825.slice/crio-02d873af4bd7c493691d31d66484303530d20d856e2849dfc3f8894021333aba WatchSource:0}: Error finding container 02d873af4bd7c493691d31d66484303530d20d856e2849dfc3f8894021333aba: Status 404 returned error can't find the container with id 02d873af4bd7c493691d31d66484303530d20d856e2849dfc3f8894021333aba Oct 30 06:01:00.235323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:00.235254 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg" event={"ID":"8f6114c0-230e-40ef-9cb5-60f49d0f4825","Type":"ContainerStarted","Data":"02d873af4bd7c493691d31d66484303530d20d856e2849dfc3f8894021333aba"} Oct 30 06:01:02.246979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:02.246886 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg" event={"ID":"8f6114c0-230e-40ef-9cb5-60f49d0f4825","Type":"ContainerStarted","Data":"2c1620fb078498637c16ce5b139ef311702a6a5de6d282bfa4aabf39adde8613"} Oct 30 06:01:02.246979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:02.246957 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg" Oct 30 06:01:02.310292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:02.310226 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg" podStartSLOduration=1.65902696 podStartE2EDuration="4.310211097s" totalImagesPullingTime="2.651184137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:02.308578391 +0000 UTC m=+476.304535401" watchObservedRunningTime="2026-10-30 06:01:02.310211097 +0000 UTC m=+476.306168106" Oct 30 06:01:07.392377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:07.392343 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7"] Oct 30 06:01:07.397444 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:07.397422 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:01:07.453624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:07.453592 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v9h79\"" Oct 30 06:01:07.455492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:07.455470 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7"] Oct 30 06:01:07.486608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:07.486573 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkpbq\" (UniqueName: \"kubernetes.io/projected/ba380ad4-2195-4cfc-b197-d1568adc5140-kube-api-access-mkpbq\") pod \"kuadrant-operator-controller-manager-678d87766f-hbpn7\" (UID: \"ba380ad4-2195-4cfc-b197-d1568adc5140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:01:07.486790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:07.486642 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba380ad4-2195-4cfc-b197-d1568adc5140-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hbpn7\" (UID: \"ba380ad4-2195-4cfc-b197-d1568adc5140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:01:07.588126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:07.588091 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkpbq\" (UniqueName: \"kubernetes.io/projected/ba380ad4-2195-4cfc-b197-d1568adc5140-kube-api-access-mkpbq\") pod \"kuadrant-operator-controller-manager-678d87766f-hbpn7\" (UID: \"ba380ad4-2195-4cfc-b197-d1568adc5140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:01:07.588298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:07.588146 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba380ad4-2195-4cfc-b197-d1568adc5140-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hbpn7\" (UID: \"ba380ad4-2195-4cfc-b197-d1568adc5140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:01:07.588524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:07.588506 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba380ad4-2195-4cfc-b197-d1568adc5140-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hbpn7\" (UID: \"ba380ad4-2195-4cfc-b197-d1568adc5140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:01:07.637268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:07.637232 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkpbq\" (UniqueName: \"kubernetes.io/projected/ba380ad4-2195-4cfc-b197-d1568adc5140-kube-api-access-mkpbq\") pod \"kuadrant-operator-controller-manager-678d87766f-hbpn7\" (UID: \"ba380ad4-2195-4cfc-b197-d1568adc5140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:01:07.710223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:07.710151 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:01:08.059852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:08.059770 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7"] Oct 30 06:01:08.064051 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:01:08.064027 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba380ad4_2195_4cfc_b197_d1568adc5140.slice/crio-cf22df4749c04688fded6b85e85a294c7cd3ef41b8cfde78a5714368c39042e3 WatchSource:0}: Error finding container cf22df4749c04688fded6b85e85a294c7cd3ef41b8cfde78a5714368c39042e3: Status 404 returned error can't find the container with id cf22df4749c04688fded6b85e85a294c7cd3ef41b8cfde78a5714368c39042e3 Oct 30 06:01:08.266669 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:01:08.266632 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:01:08.275248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:08.275208 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" event={"ID":"ba380ad4-2195-4cfc-b197-d1568adc5140","Type":"ContainerStarted","Data":"cf22df4749c04688fded6b85e85a294c7cd3ef41b8cfde78a5714368c39042e3"} Oct 30 06:01:10.520528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:10.520490 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq"] Oct 30 06:01:10.523996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:10.523975 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq" Oct 30 06:01:10.544340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:10.544298 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-5zd2k\"" Oct 30 06:01:10.558323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:10.558284 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq"] Oct 30 06:01:10.616037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:10.615996 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bvz\" (UniqueName: \"kubernetes.io/projected/937866c0-656c-45fb-bec7-b19124c11c4e-kube-api-access-f7bvz\") pod \"authorino-operator-7b7bb4bc77-9s8vq\" (UID: \"937866c0-656c-45fb-bec7-b19124c11c4e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq" Oct 30 06:01:10.717050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:10.717006 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bvz\" (UniqueName: \"kubernetes.io/projected/937866c0-656c-45fb-bec7-b19124c11c4e-kube-api-access-f7bvz\") pod \"authorino-operator-7b7bb4bc77-9s8vq\" (UID: \"937866c0-656c-45fb-bec7-b19124c11c4e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq" Oct 30 06:01:10.742200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:10.742166 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bvz\" (UniqueName: \"kubernetes.io/projected/937866c0-656c-45fb-bec7-b19124c11c4e-kube-api-access-f7bvz\") pod \"authorino-operator-7b7bb4bc77-9s8vq\" (UID: \"937866c0-656c-45fb-bec7-b19124c11c4e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq" Oct 30 06:01:10.838768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:10.838674 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq" Oct 30 06:01:11.150918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:11.150882 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq"] Oct 30 06:01:11.154994 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:01:11.154964 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937866c0_656c_45fb_bec7_b19124c11c4e.slice/crio-93a32bb96fb7aefaf9fe75242bb528e532aabb86e77254cd2c83b5d89a885f28 WatchSource:0}: Error finding container 93a32bb96fb7aefaf9fe75242bb528e532aabb86e77254cd2c83b5d89a885f28: Status 404 returned error can't find the container with id 93a32bb96fb7aefaf9fe75242bb528e532aabb86e77254cd2c83b5d89a885f28 Oct 30 06:01:11.294033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:11.293999 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq" event={"ID":"937866c0-656c-45fb-bec7-b19124c11c4e","Type":"ContainerStarted","Data":"93a32bb96fb7aefaf9fe75242bb528e532aabb86e77254cd2c83b5d89a885f28"} Oct 30 06:01:13.171517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.171488 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t"] Oct 30 06:01:13.175442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.175419 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.195884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.195860 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zn74q\"" Oct 30 06:01:13.212840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.212807 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t"] Oct 30 06:01:13.254213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.254182 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thdmg" Oct 30 06:01:13.307777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.307742 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq" event={"ID":"937866c0-656c-45fb-bec7-b19124c11c4e","Type":"ContainerStarted","Data":"c1fcdac1d1c105ef9e817a8128e69200f87046b39a90c91fc8eb327f090e3fbc"} Oct 30 06:01:13.307998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.307800 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq" Oct 30 06:01:13.309620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.309593 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" event={"ID":"ba380ad4-2195-4cfc-b197-d1568adc5140","Type":"ContainerStarted","Data":"1a5cfba7280c2a840b5c88950a1637330af9d5e2aff650eaf6be6631c180a391"} Oct 30 06:01:13.309733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.309648 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:01:13.341247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.341211 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.341428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.341274 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea443fd-b42f-4897-9731-8354eb3ca6db-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.341476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.341439 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.341511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.341479 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl6wp\" (UniqueName: \"kubernetes.io/projected/cea443fd-b42f-4897-9731-8354eb3ca6db-kube-api-access-cl6wp\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.341562 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.341543 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.341628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.341605 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.341664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.341640 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.341697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.341661 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.341733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.341702 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.375400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.375339 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq" podStartSLOduration=2.05043151 podStartE2EDuration="3.375300386s" totalImagesPullingTime="1.324868876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:13.373605831 +0000 UTC m=+487.369562842" watchObservedRunningTime="2026-10-30 06:01:13.375300386 +0000 UTC m=+487.371257395" Oct 30 06:01:13.419434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.419376 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" podStartSLOduration=2.008744583 podStartE2EDuration="6.419363498s" totalImagesPullingTime="4.410618915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:13.417436496 +0000 UTC m=+487.413393506" watchObservedRunningTime="2026-10-30 06:01:13.419363498 +0000 UTC m=+487.415320507" Oct 30 06:01:13.442567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.442487 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea443fd-b42f-4897-9731-8354eb3ca6db-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.442567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.442528 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.442567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.442546 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl6wp\" (UniqueName: \"kubernetes.io/projected/cea443fd-b42f-4897-9731-8354eb3ca6db-kube-api-access-cl6wp\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.442567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.442567 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.442909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.442629 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.442909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.442756 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.442909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.442786 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.442909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.442821 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.443105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.442910 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.443105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.442960 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.443105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.443012 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.443292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.443272 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.443387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.443366 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea443fd-b42f-4897-9731-8354eb3ca6db-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.443469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.443448 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.445118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.445097 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.445512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.445493 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.467127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.467104 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea443fd-b42f-4897-9731-8354eb3ca6db-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.467593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.467573 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl6wp\" (UniqueName: \"kubernetes.io/projected/cea443fd-b42f-4897-9731-8354eb3ca6db-kube-api-access-cl6wp\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t\" (UID: \"cea443fd-b42f-4897-9731-8354eb3ca6db\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.488281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.488257 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:13.852819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.852729 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t"] Oct 30 06:01:13.857147 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:01:13.857113 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea443fd_b42f_4897_9731_8354eb3ca6db.slice/crio-c3a251ef4074ff96a0540e4bc30e0e1ab2c54c467767c462b3c57802985eb102 WatchSource:0}: Error finding container c3a251ef4074ff96a0540e4bc30e0e1ab2c54c467767c462b3c57802985eb102: Status 404 returned error can't find the container with id c3a251ef4074ff96a0540e4bc30e0e1ab2c54c467767c462b3c57802985eb102 Oct 30 06:01:13.859529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.859486 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:13.859622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.859576 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:13.859622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:13.859616 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:14.093541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.093509 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql"] Oct 30 06:01:14.097233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.097214 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql" Oct 30 06:01:14.120628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.120599 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2q9zx\"" Oct 30 06:01:14.122153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.122123 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql"] Oct 30 06:01:14.149221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.149190 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9qp8\" (UniqueName: \"kubernetes.io/projected/f611b837-f97a-4ff0-bc4c-3119f5a8d9ca-kube-api-access-m9qp8\") pod \"dns-operator-controller-manager-8557cbdb84-g5mql\" (UID: \"f611b837-f97a-4ff0-bc4c-3119f5a8d9ca\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql" Oct 30 06:01:14.161099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.161074 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 06:01:14.250385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.250354 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9qp8\" (UniqueName: \"kubernetes.io/projected/f611b837-f97a-4ff0-bc4c-3119f5a8d9ca-kube-api-access-m9qp8\") pod \"dns-operator-controller-manager-8557cbdb84-g5mql\" (UID: \"f611b837-f97a-4ff0-bc4c-3119f5a8d9ca\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql" Oct 30 06:01:14.285236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.285199 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9qp8\" (UniqueName: \"kubernetes.io/projected/f611b837-f97a-4ff0-bc4c-3119f5a8d9ca-kube-api-access-m9qp8\") pod \"dns-operator-controller-manager-8557cbdb84-g5mql\" (UID: \"f611b837-f97a-4ff0-bc4c-3119f5a8d9ca\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql" Oct 30 06:01:14.316502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.316470 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" event={"ID":"cea443fd-b42f-4897-9731-8354eb3ca6db","Type":"ContainerStarted","Data":"9e7c9927b452c6af179253f20fabdfe39cea4805be0f2d20633a3e286e870f5c"} Oct 30 06:01:14.316676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.316523 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" event={"ID":"cea443fd-b42f-4897-9731-8354eb3ca6db","Type":"ContainerStarted","Data":"c3a251ef4074ff96a0540e4bc30e0e1ab2c54c467767c462b3c57802985eb102"} Oct 30 06:01:14.379529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.379469 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" podStartSLOduration=1.379451966 podStartE2EDuration="1.379451966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:14.378025898 +0000 UTC m=+488.373982907" watchObservedRunningTime="2026-10-30 06:01:14.379451966 +0000 UTC m=+488.375408976" Oct 30 06:01:14.411205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.411170 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql" Oct 30 06:01:14.488740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.488710 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:14.493635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.493609 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:14.729126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:14.729046 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql"] Oct 30 06:01:14.738598 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:01:14.738573 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf611b837_f97a_4ff0_bc4c_3119f5a8d9ca.slice/crio-fd42e609ae54a315603e6af0a71f330169f1fe3f7480b466f938369a657ae634 WatchSource:0}: Error finding container fd42e609ae54a315603e6af0a71f330169f1fe3f7480b466f938369a657ae634: Status 404 returned error can't find the container with id fd42e609ae54a315603e6af0a71f330169f1fe3f7480b466f938369a657ae634 Oct 30 06:01:15.323136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:15.323106 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql" event={"ID":"f611b837-f97a-4ff0-bc4c-3119f5a8d9ca","Type":"ContainerStarted","Data":"fd42e609ae54a315603e6af0a71f330169f1fe3f7480b466f938369a657ae634"} Oct 30 06:01:15.323581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:15.323339 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:15.324270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:15.324251 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t" Oct 30 06:01:17.334864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:17.334825 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql" event={"ID":"f611b837-f97a-4ff0-bc4c-3119f5a8d9ca","Type":"ContainerStarted","Data":"e2925a5c3dbd28ea71137e47b7b92072aae6840952f5926494bf0b6127f52a75"} Oct 30 06:01:17.396008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:17.395956 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql" podStartSLOduration=1.681978062 podStartE2EDuration="3.39594213s" totalImagesPullingTime="1.713964068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:17.394843311 +0000 UTC m=+491.390800322" watchObservedRunningTime="2026-10-30 06:01:17.39594213 +0000 UTC m=+491.391899140" Oct 30 06:01:18.339624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:18.339594 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql" Oct 30 06:01:20.309951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.309920 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hmcmc"] Oct 30 06:01:20.313441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.313415 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hmcmc" Oct 30 06:01:20.334261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.334238 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5khf9\"" Oct 30 06:01:20.338824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.338799 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hmcmc"] Oct 30 06:01:20.403039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.403003 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:01:20.406536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.406515 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" Oct 30 06:01:20.406794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.406768 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x4kt\" (UniqueName: \"kubernetes.io/projected/0412efee-e49f-40b4-aa1e-b0cf85bfeb83-kube-api-access-2x4kt\") pod \"authorino-57f7f8659f-hmcmc\" (UID: \"0412efee-e49f-40b4-aa1e-b0cf85bfeb83\") " pod="kuadrant-system/authorino-57f7f8659f-hmcmc" Oct 30 06:01:20.421681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.421659 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 06:01:20.430690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.430665 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:01:20.435813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.435790 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jh7qr\"" Oct 30 06:01:20.482546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.482510 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:01:20.507942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.507902 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x4kt\" (UniqueName: \"kubernetes.io/projected/0412efee-e49f-40b4-aa1e-b0cf85bfeb83-kube-api-access-2x4kt\") pod \"authorino-57f7f8659f-hmcmc\" (UID: \"0412efee-e49f-40b4-aa1e-b0cf85bfeb83\") " pod="kuadrant-system/authorino-57f7f8659f-hmcmc" Oct 30 06:01:20.508101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.507956 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ed04332-3bea-4244-9a83-4976f78ee337-config-file\") pod \"limitador-limitador-7c9b87d574-66g66\" (UID: \"3ed04332-3bea-4244-9a83-4976f78ee337\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" Oct 30 06:01:20.508101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.507995 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j78s\" (UniqueName: \"kubernetes.io/projected/3ed04332-3bea-4244-9a83-4976f78ee337-kube-api-access-7j78s\") pod \"limitador-limitador-7c9b87d574-66g66\" (UID: \"3ed04332-3bea-4244-9a83-4976f78ee337\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" Oct 30 06:01:20.531110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.531082 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x4kt\" (UniqueName: \"kubernetes.io/projected/0412efee-e49f-40b4-aa1e-b0cf85bfeb83-kube-api-access-2x4kt\") pod \"authorino-57f7f8659f-hmcmc\" (UID: \"0412efee-e49f-40b4-aa1e-b0cf85bfeb83\") " pod="kuadrant-system/authorino-57f7f8659f-hmcmc" Oct 30 06:01:20.608984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.608954 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ed04332-3bea-4244-9a83-4976f78ee337-config-file\") pod \"limitador-limitador-7c9b87d574-66g66\" (UID: \"3ed04332-3bea-4244-9a83-4976f78ee337\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" Oct 30 06:01:20.609155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.609014 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j78s\" (UniqueName: \"kubernetes.io/projected/3ed04332-3bea-4244-9a83-4976f78ee337-kube-api-access-7j78s\") pod \"limitador-limitador-7c9b87d574-66g66\" (UID: \"3ed04332-3bea-4244-9a83-4976f78ee337\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" Oct 30 06:01:20.609704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.609684 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ed04332-3bea-4244-9a83-4976f78ee337-config-file\") pod \"limitador-limitador-7c9b87d574-66g66\" (UID: \"3ed04332-3bea-4244-9a83-4976f78ee337\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" Oct 30 06:01:20.634847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.634815 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j78s\" (UniqueName: \"kubernetes.io/projected/3ed04332-3bea-4244-9a83-4976f78ee337-kube-api-access-7j78s\") pod \"limitador-limitador-7c9b87d574-66g66\" (UID: \"3ed04332-3bea-4244-9a83-4976f78ee337\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" Oct 30 06:01:20.636481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.636463 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hmcmc" Oct 30 06:01:20.718871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.718843 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" Oct 30 06:01:20.937535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:20.937452 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hmcmc"] Oct 30 06:01:20.941817 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:01:20.941788 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0412efee_e49f_40b4_aa1e_b0cf85bfeb83.slice/crio-a1bc21a35ea572512ac4ad78ed74ac16ccbea39f2fcea706a9d1e443c54db3f9 WatchSource:0}: Error finding container a1bc21a35ea572512ac4ad78ed74ac16ccbea39f2fcea706a9d1e443c54db3f9: Status 404 returned error can't find the container with id a1bc21a35ea572512ac4ad78ed74ac16ccbea39f2fcea706a9d1e443c54db3f9 Oct 30 06:01:21.020588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:21.020557 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:01:21.024132 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:01:21.024104 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed04332_3bea_4244_9a83_4976f78ee337.slice/crio-1170131b2da3dbaf62341069751f0ec1eb43fd1bedaf13808894feacb7697bed WatchSource:0}: Error finding container 1170131b2da3dbaf62341069751f0ec1eb43fd1bedaf13808894feacb7697bed: Status 404 returned error can't find the container with id 1170131b2da3dbaf62341069751f0ec1eb43fd1bedaf13808894feacb7697bed Oct 30 06:01:21.355367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:21.355299 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" event={"ID":"3ed04332-3bea-4244-9a83-4976f78ee337","Type":"ContainerStarted","Data":"1170131b2da3dbaf62341069751f0ec1eb43fd1bedaf13808894feacb7697bed"} Oct 30 06:01:21.356553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:21.356531 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hmcmc" event={"ID":"0412efee-e49f-40b4-aa1e-b0cf85bfeb83","Type":"ContainerStarted","Data":"a1bc21a35ea572512ac4ad78ed74ac16ccbea39f2fcea706a9d1e443c54db3f9"} Oct 30 06:01:22.363917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:22.363877 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" event={"ID":"3ed04332-3bea-4244-9a83-4976f78ee337","Type":"ContainerStarted","Data":"0978b6a66bb8173c25869e778f49065f2c4be106df81592059de2d9aafc8970a"} Oct 30 06:01:22.364387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:22.363967 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" Oct 30 06:01:22.450782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:22.450728 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" podStartSLOduration=1.5283680130000001 podStartE2EDuration="2.450712481s" totalImagesPullingTime="922.344468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:22.448178402 +0000 UTC m=+496.444135413" watchObservedRunningTime="2026-10-30 06:01:22.450712481 +0000 UTC m=+496.446669490" Oct 30 06:01:23.370659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:23.370617 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hmcmc" event={"ID":"0412efee-e49f-40b4-aa1e-b0cf85bfeb83","Type":"ContainerStarted","Data":"f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988"} Oct 30 06:01:23.420868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:23.420814 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-hmcmc" podStartSLOduration=1.796700763 podStartE2EDuration="3.420800221s" totalImagesPullingTime="1.624099458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:23.419477214 +0000 UTC m=+497.415434229" watchObservedRunningTime="2026-10-30 06:01:23.420800221 +0000 UTC m=+497.416757230" Oct 30 06:01:24.318423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:24.318394 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9s8vq" Oct 30 06:01:24.318586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:24.318470 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:01:24.454438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:24.454407 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hmcmc"] Oct 30 06:01:25.380470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:25.380432 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-hmcmc" podUID="0412efee-e49f-40b4-aa1e-b0cf85bfeb83" containerName="authorino" containerID="cri-o://f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988" gracePeriod=30 Oct 30 06:01:25.632475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:25.632420 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hmcmc" Oct 30 06:01:25.749392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:25.749358 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0412efee-e49f-40b4-aa1e-b0cf85bfeb83-kube-api-access-2x4kt\" (UniqueName: \"kubernetes.io/projected/0412efee-e49f-40b4-aa1e-b0cf85bfeb83-kube-api-access-2x4kt\") pod \"0412efee-e49f-40b4-aa1e-b0cf85bfeb83\" (UID: \"0412efee-e49f-40b4-aa1e-b0cf85bfeb83\") " Oct 30 06:01:25.751787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:25.751754 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0412efee-e49f-40b4-aa1e-b0cf85bfeb83-kube-api-access-2x4kt" pod "0412efee-e49f-40b4-aa1e-b0cf85bfeb83" (UID: "0412efee-e49f-40b4-aa1e-b0cf85bfeb83"). InnerVolumeSpecName "kube-api-access-2x4kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:01:25.850514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:25.850482 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x4kt\" (UniqueName: \"kubernetes.io/projected/0412efee-e49f-40b4-aa1e-b0cf85bfeb83-kube-api-access-2x4kt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:01:26.386907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:26.386807 5001 generic.go:356] "Generic (PLEG): container finished" podID="0412efee-e49f-40b4-aa1e-b0cf85bfeb83" containerID="f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988" exitCode=0 Oct 30 06:01:26.386907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:26.386853 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hmcmc" Oct 30 06:01:26.386907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:26.386874 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hmcmc" event={"ID":"0412efee-e49f-40b4-aa1e-b0cf85bfeb83","Type":"ContainerDied","Data":"f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988"} Oct 30 06:01:26.387152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:26.386922 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hmcmc" event={"ID":"0412efee-e49f-40b4-aa1e-b0cf85bfeb83","Type":"ContainerDied","Data":"a1bc21a35ea572512ac4ad78ed74ac16ccbea39f2fcea706a9d1e443c54db3f9"} Oct 30 06:01:26.387152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:26.386947 5001 scope.go:122] "RemoveContainer" containerID="f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988" Oct 30 06:01:26.395813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:26.395799 5001 scope.go:122] "RemoveContainer" containerID="f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988" Oct 30 06:01:26.396087 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:01:26.396065 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988\": container with ID starting with f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988 not found: ID does not exist" containerID="f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988" Oct 30 06:01:26.396134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:26.396096 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988"} err="failed to get container status \"f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988\": rpc error: code = NotFound desc = could not find container \"f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988\": container with ID starting with f1b4898ae5329b0d47ed593b84a4902e3d87606465d959e392ef3217f5598988 not found: ID does not exist" Oct 30 06:01:26.458405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:26.458371 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hmcmc"] Oct 30 06:01:26.467900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:26.467870 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hmcmc"] Oct 30 06:01:28.114954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:28.114921 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0412efee-e49f-40b4-aa1e-b0cf85bfeb83" path="/var/lib/kubelet/pods/0412efee-e49f-40b4-aa1e-b0cf85bfeb83/volumes" Oct 30 06:01:29.347157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:29.347122 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-g5mql" Oct 30 06:01:33.372372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:01:33.372342 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-66g66" Oct 30 06:01:38.265950 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:01:38.265916 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:02:01.219082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.219043 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-786557dc8d-52sb6"] Oct 30 06:02:01.219606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.219492 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0412efee-e49f-40b4-aa1e-b0cf85bfeb83" containerName="authorino" Oct 30 06:02:01.219606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.219504 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0412efee-e49f-40b4-aa1e-b0cf85bfeb83" containerName="authorino" Oct 30 06:02:01.219606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.219572 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0412efee-e49f-40b4-aa1e-b0cf85bfeb83" containerName="authorino" Oct 30 06:02:01.224901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.224877 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:01.242876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.242838 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 06:02:01.243041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.242926 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 06:02:01.254348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.254318 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 06:02:01.254579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.254566 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-52xdc\"" Oct 30 06:02:01.254642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.254561 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 06:02:01.254642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.254626 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 06:02:01.257448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.257424 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-786557dc8d-52sb6"] Oct 30 06:02:01.267419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.267395 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-metrics-certs\") pod \"maas-controller-786557dc8d-52sb6\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:01.267582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.267423 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfns5\" (UniqueName: \"kubernetes.io/projected/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-kube-api-access-tfns5\") pod \"maas-controller-786557dc8d-52sb6\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:01.267645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.267575 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-webhook-cert\") pod \"maas-controller-786557dc8d-52sb6\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:01.368289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.368252 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-metrics-certs\") pod \"maas-controller-786557dc8d-52sb6\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:01.368289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.368293 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfns5\" (UniqueName: \"kubernetes.io/projected/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-kube-api-access-tfns5\") pod \"maas-controller-786557dc8d-52sb6\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:01.368556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.368398 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-webhook-cert\") pod \"maas-controller-786557dc8d-52sb6\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:01.371091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.371059 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-metrics-certs\") pod \"maas-controller-786557dc8d-52sb6\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:01.371227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.371071 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-webhook-cert\") pod \"maas-controller-786557dc8d-52sb6\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:01.393361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.393298 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfns5\" (UniqueName: \"kubernetes.io/projected/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-kube-api-access-tfns5\") pod \"maas-controller-786557dc8d-52sb6\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:01.538608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.538514 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:01.847174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:01.847097 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-786557dc8d-52sb6"] Oct 30 06:02:01.851017 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:02:01.850986 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcbc525_ec5a_4af6_939e_59b1cf2325cf.slice/crio-e4f56c608a7452d26823ebfdc505f796cfacfeb6f1ed0c95ee67efa3b8e700fb WatchSource:0}: Error finding container e4f56c608a7452d26823ebfdc505f796cfacfeb6f1ed0c95ee67efa3b8e700fb: Status 404 returned error can't find the container with id e4f56c608a7452d26823ebfdc505f796cfacfeb6f1ed0c95ee67efa3b8e700fb Oct 30 06:02:02.567735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:02.567701 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-52sb6" event={"ID":"9bcbc525-ec5a-4af6-939e-59b1cf2325cf","Type":"ContainerStarted","Data":"e4f56c608a7452d26823ebfdc505f796cfacfeb6f1ed0c95ee67efa3b8e700fb"} Oct 30 06:02:04.580065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:04.579962 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-52sb6" event={"ID":"9bcbc525-ec5a-4af6-939e-59b1cf2325cf","Type":"ContainerStarted","Data":"2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763"} Oct 30 06:02:04.580492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:04.580145 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:04.636955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:04.636905 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-786557dc8d-52sb6" podStartSLOduration=1.20298901 podStartE2EDuration="3.636890231s" totalImagesPullingTime="2.433901221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:04.63591895 +0000 UTC m=+538.631875960" watchObservedRunningTime="2026-10-30 06:02:04.636890231 +0000 UTC m=+538.632847241" Oct 30 06:02:08.265474 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:02:08.265441 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:02:15.588193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:15.588165 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:16.770051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:16.770014 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-786557dc8d-52sb6"] Oct 30 06:02:16.770570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:16.770249 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-786557dc8d-52sb6" podUID="9bcbc525-ec5a-4af6-939e-59b1cf2325cf" containerName="manager" containerID="cri-o://2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763" gracePeriod=30 Oct 30 06:02:17.025919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.025861 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:17.114485 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.114448 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-metrics-certs\") pod \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " Oct 30 06:02:17.114719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.114566 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-webhook-cert\") pod \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " Oct 30 06:02:17.114719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.114606 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-kube-api-access-tfns5\" (UniqueName: \"kubernetes.io/projected/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-kube-api-access-tfns5\") pod \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\" (UID: \"9bcbc525-ec5a-4af6-939e-59b1cf2325cf\") " Oct 30 06:02:17.121330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.121180 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-metrics-certs" pod "9bcbc525-ec5a-4af6-939e-59b1cf2325cf" (UID: "9bcbc525-ec5a-4af6-939e-59b1cf2325cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:17.121330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.121274 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-kube-api-access-tfns5" pod "9bcbc525-ec5a-4af6-939e-59b1cf2325cf" (UID: "9bcbc525-ec5a-4af6-939e-59b1cf2325cf"). InnerVolumeSpecName "kube-api-access-tfns5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:17.121525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.121361 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-webhook-cert" pod "9bcbc525-ec5a-4af6-939e-59b1cf2325cf" (UID: "9bcbc525-ec5a-4af6-939e-59b1cf2325cf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:17.215492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.215461 5001 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-webhook-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:02:17.215492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.215487 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfns5\" (UniqueName: \"kubernetes.io/projected/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-kube-api-access-tfns5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:02:17.215492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.215496 5001 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcbc525-ec5a-4af6-939e-59b1cf2325cf-metrics-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:02:17.645173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.645137 5001 generic.go:356] "Generic (PLEG): container finished" podID="9bcbc525-ec5a-4af6-939e-59b1cf2325cf" containerID="2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763" exitCode=0 Oct 30 06:02:17.645368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.645188 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-52sb6" Oct 30 06:02:17.645368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.645222 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-52sb6" event={"ID":"9bcbc525-ec5a-4af6-939e-59b1cf2325cf","Type":"ContainerDied","Data":"2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763"} Oct 30 06:02:17.645368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.645284 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-52sb6" event={"ID":"9bcbc525-ec5a-4af6-939e-59b1cf2325cf","Type":"ContainerDied","Data":"e4f56c608a7452d26823ebfdc505f796cfacfeb6f1ed0c95ee67efa3b8e700fb"} Oct 30 06:02:17.645368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.645324 5001 scope.go:122] "RemoveContainer" containerID="2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763" Oct 30 06:02:17.654897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.654879 5001 scope.go:122] "RemoveContainer" containerID="2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763" Oct 30 06:02:17.655173 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:02:17.655151 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763\": container with ID starting with 2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763 not found: ID does not exist" containerID="2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763" Oct 30 06:02:17.655223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.655185 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763"} err="failed to get container status \"2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763\": rpc error: code = NotFound desc = could not find container \"2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763\": container with ID starting with 2681d85d074bed87ea2bc1a0ef8347e6abfc8777ca732c775b4a7e865236d763 not found: ID does not exist" Oct 30 06:02:17.695720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.695691 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-786557dc8d-52sb6"] Oct 30 06:02:17.706915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:17.706890 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-786557dc8d-52sb6"] Oct 30 06:02:18.115659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:18.115625 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bcbc525-ec5a-4af6-939e-59b1cf2325cf" path="/var/lib/kubelet/pods/9bcbc525-ec5a-4af6-939e-59b1cf2325cf/volumes" Oct 30 06:02:19.181941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.181907 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz"] Oct 30 06:02:19.182465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.182285 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9bcbc525-ec5a-4af6-939e-59b1cf2325cf" containerName="manager" Oct 30 06:02:19.182465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.182295 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bcbc525-ec5a-4af6-939e-59b1cf2325cf" containerName="manager" Oct 30 06:02:19.182465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.182370 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bcbc525-ec5a-4af6-939e-59b1cf2325cf" containerName="manager" Oct 30 06:02:19.185268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.185253 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.201238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.201218 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 06:02:19.201379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.201294 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 06:02:19.201441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.201382 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 06:02:19.212780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.212757 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dbg4t\"" Oct 30 06:02:19.236878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.236850 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz"] Oct 30 06:02:19.334640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.334598 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mj6\" (UniqueName: \"kubernetes.io/projected/e1d1dda1-333e-4206-9933-ffe9acdc95c7-kube-api-access-p7mj6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.334828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.334650 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.334828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.334743 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1d1dda1-333e-4206-9933-ffe9acdc95c7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.334956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.334837 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.334956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.334886 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d1dda1-333e-4206-9933-ffe9acdc95c7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.335044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.334967 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.335044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.334989 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.435964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.435860 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.435964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.435906 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.436204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.435975 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mj6\" (UniqueName: \"kubernetes.io/projected/e1d1dda1-333e-4206-9933-ffe9acdc95c7-kube-api-access-p7mj6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.436204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.436024 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.436204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.436085 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1d1dda1-333e-4206-9933-ffe9acdc95c7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.436204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.436113 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.436204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.436156 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d1dda1-333e-4206-9933-ffe9acdc95c7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.436758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.436732 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.438586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.438558 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.438690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.438618 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1d1dda1-333e-4206-9933-ffe9acdc95c7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.438848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.438826 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d1dda1-333e-4206-9933-ffe9acdc95c7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.438913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.438899 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.461867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.461836 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d1dda1-333e-4206-9933-ffe9acdc95c7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.462029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.461985 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mj6\" (UniqueName: \"kubernetes.io/projected/e1d1dda1-333e-4206-9933-ffe9acdc95c7-kube-api-access-p7mj6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scslz\" (UID: \"e1d1dda1-333e-4206-9933-ffe9acdc95c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.496430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.496402 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:19.829538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:19.829466 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz"] Oct 30 06:02:19.846058 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:02:19.846027 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d1dda1_333e_4206_9933_ffe9acdc95c7.slice/crio-c0d0ce6c59bb0cdcae14d37aa614c6fb7241b829aca9fa138c1346689adde40b WatchSource:0}: Error finding container c0d0ce6c59bb0cdcae14d37aa614c6fb7241b829aca9fa138c1346689adde40b: Status 404 returned error can't find the container with id c0d0ce6c59bb0cdcae14d37aa614c6fb7241b829aca9fa138c1346689adde40b Oct 30 06:02:20.665840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:20.665807 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" event={"ID":"e1d1dda1-333e-4206-9933-ffe9acdc95c7","Type":"ContainerStarted","Data":"c0d0ce6c59bb0cdcae14d37aa614c6fb7241b829aca9fa138c1346689adde40b"} Oct 30 06:02:22.596660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:22.596628 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:22.596945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:22.596694 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:22.678545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:22.678498 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" event={"ID":"e1d1dda1-333e-4206-9933-ffe9acdc95c7","Type":"ContainerStarted","Data":"cfca1eb30a04c8ac0fda8fc1a1220052bbe41a8ba5dd44d6f38391fe629e1299"} Oct 30 06:02:22.678708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:22.678625 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:22.758415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:22.758361 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" podStartSLOduration=1.010542344 podStartE2EDuration="3.758347051s" totalImagesPullingTime="2.747804707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:22.745070836 +0000 UTC m=+556.741027847" watchObservedRunningTime="2026-10-30 06:02:22.758347051 +0000 UTC m=+556.754304060" Oct 30 06:02:23.685136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:23.685097 5001 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-scslz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:02:23.685591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:23.685157 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" podUID="e1d1dda1-333e-4206-9933-ffe9acdc95c7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:02:26.684053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:26.684016 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scslz" Oct 30 06:02:37.979964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:37.979930 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:02:38.287862 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:02:38.287514 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:02:46.823488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:46.823376 5001 generic.go:356] "Generic (PLEG): container finished" podID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerID="b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b" exitCode=0 Oct 30 06:02:46.823488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:46.823453 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" event={"ID":"1c68d683-85ff-4411-bf67-d0f99a0c8d41","Type":"ContainerDied","Data":"b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b"} Oct 30 06:02:46.824394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:46.823868 5001 scope.go:122] "RemoveContainer" containerID="b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b" Oct 30 06:02:46.825021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:46.824994 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:46.825080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:46.825060 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:46.825115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:46.825092 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:47.830262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:47.830221 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" event={"ID":"1c68d683-85ff-4411-bf67-d0f99a0c8d41","Type":"ContainerStarted","Data":"22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16"} Oct 30 06:02:48.174332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:48.173860 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:02:48.174332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:48.174039 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l5d6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 30 06:02:48.174332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:48.174083 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 30 06:02:49.169711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:49.168757 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l5d6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 30 06:02:49.169711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:49.168821 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 30 06:02:50.048996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:50.048944 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:02:50.170770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:50.170731 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l5d6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 30 06:02:50.171412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:50.171377 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 30 06:02:51.166397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:51.166359 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l5d6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 30 06:02:51.166689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:51.166659 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 30 06:02:52.163818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:52.163775 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-l5d6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 30 06:02:52.176355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:52.164445 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 30 06:02:53.167957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:53.167929 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:02:53.168807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:02:53.168788 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:03:06.167105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:03:06.167084 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:03:06.169239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:03:06.169219 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:03:08.264504 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:03:08.264464 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:03:38.266491 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:03:38.266452 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:04:08.266923 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:04:08.266886 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:04:38.274670 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:04:38.274577 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:04:52.105826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:04:52.105789 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:04:59.549841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:04:59.549802 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:05:04.265109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:04.265071 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:05:07.315641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.315600 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:05:07.745936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.745900 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5vzlj"] Oct 30 06:05:07.749224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.749190 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.773678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.773640 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5vzlj"] Oct 30 06:05:07.779008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.778980 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8c689\"" Oct 30 06:05:07.779008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.778980 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 06:05:07.836987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.836951 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.837208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.837024 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwljt\" (UniqueName: \"kubernetes.io/projected/efd5ce32-5bd7-4589-8f5b-30f580257a13-kube-api-access-mwljt\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.837208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.837072 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd5ce32-5bd7-4589-8f5b-30f580257a13-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.837208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.837192 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-home\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.837431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.837223 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.837431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.837256 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.844563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.844538 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 06:05:07.844563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.844549 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 06:05:07.938229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.938194 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd5ce32-5bd7-4589-8f5b-30f580257a13-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.938458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.938268 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-home\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.938458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.938290 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.938762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.938736 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-home\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.938940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.938876 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.938940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.938927 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.939075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.939003 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwljt\" (UniqueName: \"kubernetes.io/projected/efd5ce32-5bd7-4589-8f5b-30f580257a13-kube-api-access-mwljt\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.939534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.939509 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.939647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.939622 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.941345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.941320 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd5ce32-5bd7-4589-8f5b-30f580257a13-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.941601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.941579 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd5ce32-5bd7-4589-8f5b-30f580257a13-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:07.963190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:07.963154 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwljt\" (UniqueName: \"kubernetes.io/projected/efd5ce32-5bd7-4589-8f5b-30f580257a13-kube-api-access-mwljt\") pod \"echo-simulator-kserve-64b6f8f96c-5vzlj\" (UID: \"efd5ce32-5bd7-4589-8f5b-30f580257a13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:08.064965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:08.064870 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:08.276808 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:05:08.276661 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:05:08.372190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:08.372163 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5vzlj"] Oct 30 06:05:08.374661 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:05:08.374603 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd5ce32_5bd7_4589_8f5b_30f580257a13.slice/crio-c06a04256e57f12761ed35d4b6823f255ed45127270b1249b81b7d9e8a7a13cb WatchSource:0}: Error finding container c06a04256e57f12761ed35d4b6823f255ed45127270b1249b81b7d9e8a7a13cb: Status 404 returned error can't find the container with id c06a04256e57f12761ed35d4b6823f255ed45127270b1249b81b7d9e8a7a13cb Oct 30 06:05:08.376213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:08.376191 5001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:05:08.551435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:08.551399 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" event={"ID":"efd5ce32-5bd7-4589-8f5b-30f580257a13","Type":"ContainerStarted","Data":"c06a04256e57f12761ed35d4b6823f255ed45127270b1249b81b7d9e8a7a13cb"} Oct 30 06:05:09.656376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.656328 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl"] Oct 30 06:05:09.660422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.660396 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.681149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.681070 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 06:05:09.695031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.694997 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl"] Oct 30 06:05:09.753144 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.753101 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.753144 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.753148 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncr7b\" (UniqueName: \"kubernetes.io/projected/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-kube-api-access-ncr7b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.753463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.753261 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.753463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.753375 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.753463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.753412 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.753576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.753474 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.835293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.835260 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk"] Oct 30 06:05:09.838484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.838467 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.855360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.854265 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.855360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.854326 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.855360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.854355 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.855360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.854384 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.855360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.854812 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.855360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.855153 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.855772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.855751 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.855848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.855804 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpfx4\" (UniqueName: \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-kube-api-access-bpfx4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.855917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.855858 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.855975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.855955 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.856048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.856022 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.856103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.856079 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d343287d-c9ed-4e9b-9891-438d1007cccb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.856157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.856111 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.857410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.856607 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.857410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.857183 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.857410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.857218 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncr7b\" (UniqueName: \"kubernetes.io/projected/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-kube-api-access-ncr7b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.857410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.857252 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.857636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.857485 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.858910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.858889 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.860516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.860494 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.877513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.877433 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk"] Oct 30 06:05:09.893441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.893413 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncr7b\" (UniqueName: \"kubernetes.io/projected/cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0-kube-api-access-ncr7b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl\" (UID: \"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.958585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.958561 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.958712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.958603 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.958712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.958631 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.958712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.958664 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfx4\" (UniqueName: \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-kube-api-access-bpfx4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.958878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.958709 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.958878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.958743 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.958878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.958785 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d343287d-c9ed-4e9b-9891-438d1007cccb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.958878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.958816 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.959091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.958910 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.959091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.958987 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.959091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.959052 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.959261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.959207 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.959359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.959337 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.959612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.959593 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d343287d-c9ed-4e9b-9891-438d1007cccb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.961239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.961219 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.961546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.961525 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.978224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.978191 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:09.983386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.983364 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpfx4\" (UniqueName: \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-kube-api-access-bpfx4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:09.983735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:09.983713 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6g2sk\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:10.157558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:10.157464 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:10.288882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:10.288852 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl"] Oct 30 06:05:10.291195 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:05:10.291162 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5f3ec1_3f05_40ba_a14c_b6ddc6a00db0.slice/crio-800508ec0f3bdfd49a2d7563aae5ee35601b1e46d1440644919e5f316b3271b9 WatchSource:0}: Error finding container 800508ec0f3bdfd49a2d7563aae5ee35601b1e46d1440644919e5f316b3271b9: Status 404 returned error can't find the container with id 800508ec0f3bdfd49a2d7563aae5ee35601b1e46d1440644919e5f316b3271b9 Oct 30 06:05:10.487511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:10.487482 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk"] Oct 30 06:05:10.489956 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:05:10.489926 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd343287d_c9ed_4e9b_9891_438d1007cccb.slice/crio-534dcccbb140f7fa36bd7a738e7c8af999fbc58e2305c37293508e370149b211 WatchSource:0}: Error finding container 534dcccbb140f7fa36bd7a738e7c8af999fbc58e2305c37293508e370149b211: Status 404 returned error can't find the container with id 534dcccbb140f7fa36bd7a738e7c8af999fbc58e2305c37293508e370149b211 Oct 30 06:05:10.492167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:10.492130 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:10.492284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:10.492209 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:10.492284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:10.492251 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:10.574290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:10.574248 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" event={"ID":"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0","Type":"ContainerStarted","Data":"800508ec0f3bdfd49a2d7563aae5ee35601b1e46d1440644919e5f316b3271b9"} Oct 30 06:05:10.576597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:10.576524 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" event={"ID":"efd5ce32-5bd7-4589-8f5b-30f580257a13","Type":"ContainerStarted","Data":"2715d020658a09067b38080c09d12c991720bf3fdfca1586e7ad10e044ffe3b8"} Oct 30 06:05:10.578864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:10.578836 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" event={"ID":"d343287d-c9ed-4e9b-9891-438d1007cccb","Type":"ContainerStarted","Data":"36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e"} Oct 30 06:05:10.578978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:10.578883 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" event={"ID":"d343287d-c9ed-4e9b-9891-438d1007cccb","Type":"ContainerStarted","Data":"534dcccbb140f7fa36bd7a738e7c8af999fbc58e2305c37293508e370149b211"} Oct 30 06:05:10.626917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:10.626035 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" podStartSLOduration=2.111732411 podStartE2EDuration="3.626014309s" totalImagesPullingTime="1.514281898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:10.622728658 +0000 UTC m=+724.618685671" watchObservedRunningTime="2026-10-30 06:05:10.626014309 +0000 UTC m=+724.621971320" Oct 30 06:05:11.065512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:11.065422 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:11.174021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:11.173946 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:11.176589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:11.176195 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6g2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:05:11.176589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:11.176256 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 06:05:12.158336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:12.158214 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6g2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:05:12.158336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:12.158267 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 06:05:12.607440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:12.607387 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" event={"ID":"cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0","Type":"ContainerStarted","Data":"6841ac4ae5c7f272a0d30de0eb6400ac11f708831812003482df9dd0b8f99e51"} Oct 30 06:05:12.651507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:12.651442 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" podStartSLOduration=1.852057137 podStartE2EDuration="3.651421748s" totalImagesPullingTime="1.799364611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:12.650576398 +0000 UTC m=+726.646533427" watchObservedRunningTime="2026-10-30 06:05:12.651421748 +0000 UTC m=+726.647378826" Oct 30 06:05:12.655335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:12.651861 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podStartSLOduration=3.651850112 podStartE2EDuration="3.651850112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:10.677478623 +0000 UTC m=+724.673435634" watchObservedRunningTime="2026-10-30 06:05:12.651850112 +0000 UTC m=+726.647807119" Oct 30 06:05:12.978809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:12.978712 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:13.182636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:13.182583 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6g2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:05:13.183377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:13.183270 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 06:05:14.169747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:14.169711 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6g2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:05:14.170027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:14.169998 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 06:05:14.181285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:14.181244 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:05:15.165726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:15.165689 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6g2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:05:15.166356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:15.166294 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 06:05:16.165798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:16.165758 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6g2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:05:16.166461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:16.166428 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 06:05:17.157868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:17.157825 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6g2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:05:17.158081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:17.157896 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 06:05:18.065775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:18.065739 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:18.079320 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:18.079269 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:18.656980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:18.656947 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-5vzlj" Oct 30 06:05:19.158446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:19.158407 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6g2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:05:19.158850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:19.158459 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:19.978934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:19.978896 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:19.992991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:19.992957 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:20.199521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:20.199476 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6g2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:05:20.200007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:20.199546 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:20.500418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:20.500385 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:20.500814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:20.500788 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:20.501725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:20.501701 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:05:20.668654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:20.668623 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl" Oct 30 06:05:24.892775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.892737 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77"] Oct 30 06:05:24.896422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.896405 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:24.932153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.932113 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77"] Oct 30 06:05:24.991081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.991038 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:24.991271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.991170 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:24.991271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.991233 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:24.991271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.991267 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9029b845-bbdc-4a56-8776-23996797c555-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:24.991428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.991286 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9029b845-bbdc-4a56-8776-23996797c555-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:24.991428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.991315 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvc57\" (UniqueName: \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-kube-api-access-bvc57\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:24.991494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.991464 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:24.991535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.991511 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:24.991535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:24.991529 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.092979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.092943 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.092979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.092980 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093008 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093059 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093076 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093101 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9029b845-bbdc-4a56-8776-23996797c555-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093121 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9029b845-bbdc-4a56-8776-23996797c555-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093143 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvc57\" (UniqueName: \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-kube-api-access-bvc57\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093198 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093447 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093483 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093576 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093653 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.093932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.093911 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9029b845-bbdc-4a56-8776-23996797c555-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.095787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.095763 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.095980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.095960 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9029b845-bbdc-4a56-8776-23996797c555-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.117617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.117585 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.117724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.117691 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvc57\" (UniqueName: \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-kube-api-access-bvc57\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nnt77\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.210939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.210850 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:25.529648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.529621 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77"] Oct 30 06:05:25.531578 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:05:25.531550 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9029b845_bbdc_4a56_8776_23996797c555.slice/crio-de2b8faf246a7cee43b902fb1973d4504822bcca6a5df9ea04370272cf51f6f3 WatchSource:0}: Error finding container de2b8faf246a7cee43b902fb1973d4504822bcca6a5df9ea04370272cf51f6f3: Status 404 returned error can't find the container with id de2b8faf246a7cee43b902fb1973d4504822bcca6a5df9ea04370272cf51f6f3 Oct 30 06:05:25.533681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.533647 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:25.533752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.533716 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:25.533752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.533745 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:25.687530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.687500 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" event={"ID":"9029b845-bbdc-4a56-8776-23996797c555","Type":"ContainerStarted","Data":"36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32"} Oct 30 06:05:25.687686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.687559 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" event={"ID":"9029b845-bbdc-4a56-8776-23996797c555","Type":"ContainerStarted","Data":"de2b8faf246a7cee43b902fb1973d4504822bcca6a5df9ea04370272cf51f6f3"} Oct 30 06:05:25.736264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:25.736200 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" podStartSLOduration=1.73618492 podStartE2EDuration="1.73618492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:25.73494797 +0000 UTC m=+739.730904981" watchObservedRunningTime="2026-10-30 06:05:25.73618492 +0000 UTC m=+739.732141929" Oct 30 06:05:26.211671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:26.211634 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:26.213228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:26.213201 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nnt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:05:26.213361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:26.213250 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 06:05:27.211435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:27.211401 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nnt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:05:27.211625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:27.211454 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 06:05:28.211559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:28.211524 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nnt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:05:28.211969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:28.211574 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 06:05:29.211900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:29.211862 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nnt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:05:29.212292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:29.211929 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 06:05:30.211567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:30.211526 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nnt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:05:30.211757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:30.211586 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 06:05:31.211949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.211914 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nnt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:05:31.212398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.211968 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 06:05:31.606861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.606831 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b4cbf4fd4-7wnrx"] Oct 30 06:05:31.610526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.610507 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b4cbf4fd4-7wnrx" Oct 30 06:05:31.639131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.639105 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 06:05:31.639439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.639172 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5khf9\"" Oct 30 06:05:31.641540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.641517 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b4cbf4fd4-7wnrx"] Oct 30 06:05:31.650044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.650022 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmb7g\" (UniqueName: \"kubernetes.io/projected/383e8617-b682-40d9-9ad6-8f5be7ba56d2-kube-api-access-nmb7g\") pod \"authorino-b4cbf4fd4-7wnrx\" (UID: \"383e8617-b682-40d9-9ad6-8f5be7ba56d2\") " pod="kuadrant-system/authorino-b4cbf4fd4-7wnrx" Oct 30 06:05:31.650247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.650210 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/383e8617-b682-40d9-9ad6-8f5be7ba56d2-tls-cert\") pod \"authorino-b4cbf4fd4-7wnrx\" (UID: \"383e8617-b682-40d9-9ad6-8f5be7ba56d2\") " pod="kuadrant-system/authorino-b4cbf4fd4-7wnrx" Oct 30 06:05:31.751300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.751268 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/383e8617-b682-40d9-9ad6-8f5be7ba56d2-tls-cert\") pod \"authorino-b4cbf4fd4-7wnrx\" (UID: \"383e8617-b682-40d9-9ad6-8f5be7ba56d2\") " pod="kuadrant-system/authorino-b4cbf4fd4-7wnrx" Oct 30 06:05:31.751496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.751365 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmb7g\" (UniqueName: \"kubernetes.io/projected/383e8617-b682-40d9-9ad6-8f5be7ba56d2-kube-api-access-nmb7g\") pod \"authorino-b4cbf4fd4-7wnrx\" (UID: \"383e8617-b682-40d9-9ad6-8f5be7ba56d2\") " pod="kuadrant-system/authorino-b4cbf4fd4-7wnrx" Oct 30 06:05:31.754044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.754019 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/383e8617-b682-40d9-9ad6-8f5be7ba56d2-tls-cert\") pod \"authorino-b4cbf4fd4-7wnrx\" (UID: \"383e8617-b682-40d9-9ad6-8f5be7ba56d2\") " pod="kuadrant-system/authorino-b4cbf4fd4-7wnrx" Oct 30 06:05:31.776232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.776201 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmb7g\" (UniqueName: \"kubernetes.io/projected/383e8617-b682-40d9-9ad6-8f5be7ba56d2-kube-api-access-nmb7g\") pod \"authorino-b4cbf4fd4-7wnrx\" (UID: \"383e8617-b682-40d9-9ad6-8f5be7ba56d2\") " pod="kuadrant-system/authorino-b4cbf4fd4-7wnrx" Oct 30 06:05:31.939012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:31.938927 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b4cbf4fd4-7wnrx" Oct 30 06:05:32.212238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:32.212165 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nnt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:05:32.212701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:32.212231 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 06:05:32.271787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:32.271755 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b4cbf4fd4-7wnrx"] Oct 30 06:05:32.274023 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:05:32.273993 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod383e8617_b682_40d9_9ad6_8f5be7ba56d2.slice/crio-096c9ef771e8719b0dc013455c8c99994657b7002f6b38471868772340ccd2ab WatchSource:0}: Error finding container 096c9ef771e8719b0dc013455c8c99994657b7002f6b38471868772340ccd2ab: Status 404 returned error can't find the container with id 096c9ef771e8719b0dc013455c8c99994657b7002f6b38471868772340ccd2ab Oct 30 06:05:32.731423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:32.731381 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b4cbf4fd4-7wnrx" event={"ID":"383e8617-b682-40d9-9ad6-8f5be7ba56d2","Type":"ContainerStarted","Data":"096c9ef771e8719b0dc013455c8c99994657b7002f6b38471868772340ccd2ab"} Oct 30 06:05:33.741551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:33.741511 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b4cbf4fd4-7wnrx" event={"ID":"383e8617-b682-40d9-9ad6-8f5be7ba56d2","Type":"ContainerStarted","Data":"f410e41a6cd856ab1fa05e9e67b7952ba2a839b3441adfd41b1ece0046191c4f"} Oct 30 06:05:33.786331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:33.786245 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-b4cbf4fd4-7wnrx" podStartSLOduration=2.140603654 podStartE2EDuration="2.78622375s" totalImagesPullingTime="645.620096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:33.78459497 +0000 UTC m=+747.780551981" watchObservedRunningTime="2026-10-30 06:05:33.78622375 +0000 UTC m=+747.782180761" Oct 30 06:05:34.049627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:34.049544 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:34.049943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:34.049917 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:34.050928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:34.050904 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:05:38.273701 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:05:38.273662 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:05:39.941669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:39.941637 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s"] Oct 30 06:05:39.948919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:39.948893 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:39.974575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:39.974543 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s"] Oct 30 06:05:40.069347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.069283 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b9vf\" (UniqueName: \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-kube-api-access-2b9vf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.069539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.069421 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.069539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.069449 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.069539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.069482 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.069539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.069499 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.069758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.069562 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.069758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.069662 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4973ef-1d32-41f2-bc92-286112948b10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.069758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.069711 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.069856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.069792 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4973ef-1d32-41f2-bc92-286112948b10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.170875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.170831 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.170897 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.170923 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.170949 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.170995 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4973ef-1d32-41f2-bc92-286112948b10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.171097 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.171136 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4973ef-1d32-41f2-bc92-286112948b10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.171205 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9vf\" (UniqueName: \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-kube-api-access-2b9vf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.171276 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.171366 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.171569 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.171668 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.171876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.171803 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.172015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.171964 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4973ef-1d32-41f2-bc92-286112948b10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.173549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.173527 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.173772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.173756 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4973ef-1d32-41f2-bc92-286112948b10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.197216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.197131 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.197216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.197145 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9vf\" (UniqueName: \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-kube-api-access-2b9vf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-28h4s\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.262516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.262473 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:40.601358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.601290 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s"] Oct 30 06:05:40.603852 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:05:40.603826 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4973ef_1d32_41f2_bc92_286112948b10.slice/crio-1b403530a0e1dcbdcd69393bd1c3be378249285f28eecdda07ae7f095b56db56 WatchSource:0}: Error finding container 1b403530a0e1dcbdcd69393bd1c3be378249285f28eecdda07ae7f095b56db56: Status 404 returned error can't find the container with id 1b403530a0e1dcbdcd69393bd1c3be378249285f28eecdda07ae7f095b56db56 Oct 30 06:05:40.606600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.606566 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:40.606696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.606635 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:40.606696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.606661 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:40.787143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.787108 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" event={"ID":"3a4973ef-1d32-41f2-bc92-286112948b10","Type":"ContainerStarted","Data":"f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b"} Oct 30 06:05:40.787368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.787174 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" event={"ID":"3a4973ef-1d32-41f2-bc92-286112948b10","Type":"ContainerStarted","Data":"1b403530a0e1dcbdcd69393bd1c3be378249285f28eecdda07ae7f095b56db56"} Oct 30 06:05:40.835358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:40.835272 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" podStartSLOduration=1.835257118 podStartE2EDuration="1.835257118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:40.833205521 +0000 UTC m=+754.829162533" watchObservedRunningTime="2026-10-30 06:05:40.835257118 +0000 UTC m=+754.831214128" Oct 30 06:05:41.276363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:41.273387 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:41.277078 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:41.277050 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-28h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 06:05:41.277210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:41.277148 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 06:05:42.268769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:42.268729 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-28h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 06:05:42.269059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:42.269029 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 06:05:43.273334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:43.266781 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-28h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 06:05:43.273334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:43.266846 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 06:05:44.268334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:44.265748 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-28h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 06:05:44.268334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:44.265809 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 06:05:45.267799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:45.267759 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-28h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 06:05:45.268440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:45.268407 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 06:05:46.271773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:46.271731 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-28h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 06:05:46.272416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:46.272382 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 06:05:47.263967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:47.263876 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-28h4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 06:05:47.263967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:47.263940 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 06:05:48.897280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:48.897242 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:48.897786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:48.897724 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:05:48.898543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:05:48.898523 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:06:07.379300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.379256 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v"] Oct 30 06:06:07.392104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.392064 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.408134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.408102 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v"] Oct 30 06:06:07.421860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.421810 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-envoy\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.422041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.421873 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-credential-socket\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.422041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.422011 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-socket\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.422167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.422048 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83901a1f-9a22-4dbd-899d-a44f2e916a65-istiod-ca-cert\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.422167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.422114 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-podinfo\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.422273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.422194 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-token\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.422347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.422286 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-data\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.422412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.422390 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88lb\" (UniqueName: \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-kube-api-access-z88lb\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.422468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.422430 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-certs\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.431178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.431154 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-798193ae-gw-openshift-default-dockercfg-swx9j\"" Oct 30 06:06:07.522937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.522903 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-data\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.522965 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88lb\" (UniqueName: \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-kube-api-access-z88lb\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.522990 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-certs\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.523013 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-envoy\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.523169 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-credential-socket\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.523261 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-socket\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.523323 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83901a1f-9a22-4dbd-899d-a44f2e916a65-istiod-ca-cert\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.523344 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-data\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.523371 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-podinfo\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.523426 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-token\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.523351 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-certs\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.523576 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-credential-socket\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.523597 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-socket\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.523952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.523930 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83901a1f-9a22-4dbd-899d-a44f2e916a65-istiod-ca-cert\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.525792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.525774 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-envoy\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.525879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.525864 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-podinfo\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.582991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.582923 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-token\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.590339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.588082 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88lb\" (UniqueName: \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-kube-api-access-z88lb\") pod \"e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:07.714292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:07.714215 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:08.037293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:08.037214 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v"] Oct 30 06:06:08.050044 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:08.050014 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83901a1f_9a22_4dbd_899d_a44f2e916a65.slice/crio-bed72328a22686ba6abe558006443371dc1f0906ee57a80ed6b15de8e2026fc3 WatchSource:0}: Error finding container bed72328a22686ba6abe558006443371dc1f0906ee57a80ed6b15de8e2026fc3: Status 404 returned error can't find the container with id bed72328a22686ba6abe558006443371dc1f0906ee57a80ed6b15de8e2026fc3 Oct 30 06:06:08.052537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:08.052495 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:08.052647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:08.052594 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:08.052647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:08.052638 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:08.268562 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:06:08.268507 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:06:08.954454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:08.954420 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" event={"ID":"83901a1f-9a22-4dbd-899d-a44f2e916a65","Type":"ContainerStarted","Data":"3df2ec8899b98a23f8154d6b1c8b6f80166b94706f8377c139e17073c8b358aa"} Oct 30 06:06:08.954967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:08.954487 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" event={"ID":"83901a1f-9a22-4dbd-899d-a44f2e916a65","Type":"ContainerStarted","Data":"bed72328a22686ba6abe558006443371dc1f0906ee57a80ed6b15de8e2026fc3"} Oct 30 06:06:09.003487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:09.003424 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" podStartSLOduration=2.003409106 podStartE2EDuration="2.003409106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:09.001336686 +0000 UTC m=+782.997293696" watchObservedRunningTime="2026-10-30 06:06:09.003409106 +0000 UTC m=+782.999366191" Oct 30 06:06:09.714772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:09.714735 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:09.720363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:09.720335 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:09.960165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:09.960133 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:09.961229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:09.961213 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:12.127829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.127799 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v"] Oct 30 06:06:12.131736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.131718 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:12.150421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.150400 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-798193ae\"" Oct 30 06:06:12.168071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.168042 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v"] Oct 30 06:06:12.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.230754 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-798193ae-dockercfg-ql5k8\"" Oct 30 06:06:12.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.231637 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt9m4\" (UniqueName: \"kubernetes.io/projected/812f3d18-f834-4704-96cc-e830ffcca304-kube-api-access-qt9m4\") pod \"payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v\" (UID: \"812f3d18-f834-4704-96cc-e830ffcca304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:12.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.231813 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/812f3d18-f834-4704-96cc-e830ffcca304-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v\" (UID: \"812f3d18-f834-4704-96cc-e830ffcca304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:12.332904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.332860 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/812f3d18-f834-4704-96cc-e830ffcca304-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v\" (UID: \"812f3d18-f834-4704-96cc-e830ffcca304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:12.333092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.332960 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9m4\" (UniqueName: \"kubernetes.io/projected/812f3d18-f834-4704-96cc-e830ffcca304-kube-api-access-qt9m4\") pod \"payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v\" (UID: \"812f3d18-f834-4704-96cc-e830ffcca304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:12.333620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.333600 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/812f3d18-f834-4704-96cc-e830ffcca304-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v\" (UID: \"812f3d18-f834-4704-96cc-e830ffcca304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:12.360541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.360506 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9m4\" (UniqueName: \"kubernetes.io/projected/812f3d18-f834-4704-96cc-e830ffcca304-kube-api-access-qt9m4\") pod \"payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v\" (UID: \"812f3d18-f834-4704-96cc-e830ffcca304\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:12.444750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.444662 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:12.747172 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.747149 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v"] Oct 30 06:06:12.750502 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:12.750472 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812f3d18_f834_4704_96cc_e830ffcca304.slice/crio-c8f0342e8fee184625c72530412bad0fcddc0852715c0100b0b2b8e3ba927ade WatchSource:0}: Error finding container c8f0342e8fee184625c72530412bad0fcddc0852715c0100b0b2b8e3ba927ade: Status 404 returned error can't find the container with id c8f0342e8fee184625c72530412bad0fcddc0852715c0100b0b2b8e3ba927ade Oct 30 06:06:12.982994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:12.982954 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" event={"ID":"812f3d18-f834-4704-96cc-e830ffcca304","Type":"ContainerStarted","Data":"c8f0342e8fee184625c72530412bad0fcddc0852715c0100b0b2b8e3ba927ade"} Oct 30 06:06:13.961724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:13.961684 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9"] Oct 30 06:06:13.966011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:13.965983 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:13.985215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:13.985185 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9"] Oct 30 06:06:13.994708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:13.994666 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c45a2ff8\"" Oct 30 06:06:13.994708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:13.994701 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c45a2ff8-dockercfg-55szw\"" Oct 30 06:06:14.058627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:14.056464 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgdj\" (UniqueName: \"kubernetes.io/projected/83764313-b8cd-4158-b83c-5436bfc4018f-kube-api-access-8qgdj\") pod \"payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9\" (UID: \"83764313-b8cd-4158-b83c-5436bfc4018f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:14.058627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:14.056544 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83764313-b8cd-4158-b83c-5436bfc4018f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9\" (UID: \"83764313-b8cd-4158-b83c-5436bfc4018f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:14.157512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:14.157482 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgdj\" (UniqueName: \"kubernetes.io/projected/83764313-b8cd-4158-b83c-5436bfc4018f-kube-api-access-8qgdj\") pod \"payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9\" (UID: \"83764313-b8cd-4158-b83c-5436bfc4018f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:14.157685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:14.157527 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83764313-b8cd-4158-b83c-5436bfc4018f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9\" (UID: \"83764313-b8cd-4158-b83c-5436bfc4018f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:14.158259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:14.158240 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83764313-b8cd-4158-b83c-5436bfc4018f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9\" (UID: \"83764313-b8cd-4158-b83c-5436bfc4018f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:14.181870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:14.181831 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgdj\" (UniqueName: \"kubernetes.io/projected/83764313-b8cd-4158-b83c-5436bfc4018f-kube-api-access-8qgdj\") pod \"payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9\" (UID: \"83764313-b8cd-4158-b83c-5436bfc4018f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:14.284560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:14.284473 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:14.645452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:14.645418 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9"] Oct 30 06:06:14.648963 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:14.648930 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83764313_b8cd_4158_b83c_5436bfc4018f.slice/crio-33fb4fa86d795d21f6c594c4234c2ddd2dff8d9ae32c44b72e972447d4c47138 WatchSource:0}: Error finding container 33fb4fa86d795d21f6c594c4234c2ddd2dff8d9ae32c44b72e972447d4c47138: Status 404 returned error can't find the container with id 33fb4fa86d795d21f6c594c4234c2ddd2dff8d9ae32c44b72e972447d4c47138 Oct 30 06:06:14.984939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:14.984846 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:06:15.004960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.004911 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" event={"ID":"83764313-b8cd-4158-b83c-5436bfc4018f","Type":"ContainerStarted","Data":"33fb4fa86d795d21f6c594c4234c2ddd2dff8d9ae32c44b72e972447d4c47138"} Oct 30 06:06:15.005576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.005542 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt"] Oct 30 06:06:15.009523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.009499 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:15.055982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.055905 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt"] Oct 30 06:06:15.108967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.108918 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9c94fbc2\"" Oct 30 06:06:15.112710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.112675 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9c94fbc2-dockercfg-82hcn\"" Oct 30 06:06:15.171421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.167936 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt\" (UID: \"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:15.171421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.168058 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z6t2\" (UniqueName: \"kubernetes.io/projected/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-kube-api-access-6z6t2\") pod \"payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt\" (UID: \"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:15.269627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.269594 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt\" (UID: \"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:15.270094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.270068 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z6t2\" (UniqueName: \"kubernetes.io/projected/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-kube-api-access-6z6t2\") pod \"payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt\" (UID: \"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:15.270506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.270486 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt\" (UID: \"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:15.307082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.307052 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z6t2\" (UniqueName: \"kubernetes.io/projected/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-kube-api-access-6z6t2\") pod \"payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt\" (UID: \"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:15.332921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.332890 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:15.667662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:15.667628 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt"] Oct 30 06:06:15.671703 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:15.671675 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8396fa23_d5d9_4ee9_a6f9_b3cd4484b105.slice/crio-05747b6cdeffbc647c030794c30bc1e1d1a30cb432de2eb7fd4a3dcb36741976 WatchSource:0}: Error finding container 05747b6cdeffbc647c030794c30bc1e1d1a30cb432de2eb7fd4a3dcb36741976: Status 404 returned error can't find the container with id 05747b6cdeffbc647c030794c30bc1e1d1a30cb432de2eb7fd4a3dcb36741976 Oct 30 06:06:16.012315 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.012212 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" event={"ID":"812f3d18-f834-4704-96cc-e830ffcca304","Type":"ContainerStarted","Data":"d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f"} Oct 30 06:06:16.012720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.012360 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:16.014026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.013996 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" event={"ID":"83764313-b8cd-4158-b83c-5436bfc4018f","Type":"ContainerStarted","Data":"4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d"} Oct 30 06:06:16.014496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.014479 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:16.015759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.015724 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" event={"ID":"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105","Type":"ContainerStarted","Data":"d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a"} Oct 30 06:06:16.015864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.015774 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" event={"ID":"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105","Type":"ContainerStarted","Data":"05747b6cdeffbc647c030794c30bc1e1d1a30cb432de2eb7fd4a3dcb36741976"} Oct 30 06:06:16.015864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.015795 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:16.079329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.078608 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" podStartSLOduration=1.5658930660000001 podStartE2EDuration="4.078587986s" totalImagesPullingTime="2.51269492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:16.077785757 +0000 UTC m=+790.073742767" watchObservedRunningTime="2026-10-30 06:06:16.078587986 +0000 UTC m=+790.074545009" Oct 30 06:06:16.093037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.093007 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2"] Oct 30 06:06:16.096913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.096893 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:16.117864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.117839 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2"] Oct 30 06:06:16.123839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.123817 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-55140dc4-dockercfg-bn2cl\"" Oct 30 06:06:16.124907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.124879 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-55140dc4\"" Oct 30 06:06:16.189548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.189512 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v8r8\" (UniqueName: \"kubernetes.io/projected/e66f6275-5b9e-4426-be7a-003581644c67-kube-api-access-9v8r8\") pod \"payload-processing-e2e-ait-55140dc4-8b8949786-lshn2\" (UID: \"e66f6275-5b9e-4426-be7a-003581644c67\") " pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:16.189759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.189734 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66f6275-5b9e-4426-be7a-003581644c67-plugins-config-volume\") pod \"payload-processing-e2e-ait-55140dc4-8b8949786-lshn2\" (UID: \"e66f6275-5b9e-4426-be7a-003581644c67\") " pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:16.190435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.190394 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" podStartSLOduration=2.190380874 podStartE2EDuration="2.190380874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:16.18822965 +0000 UTC m=+790.184186660" watchObservedRunningTime="2026-10-30 06:06:16.190380874 +0000 UTC m=+790.186337884" Oct 30 06:06:16.191343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.191290 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" podStartSLOduration=2.466530456 podStartE2EDuration="3.191281794s" totalImagesPullingTime="724.751338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:16.137034361 +0000 UTC m=+790.132991375" watchObservedRunningTime="2026-10-30 06:06:16.191281794 +0000 UTC m=+790.187238803" Oct 30 06:06:16.291150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.291058 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v8r8\" (UniqueName: \"kubernetes.io/projected/e66f6275-5b9e-4426-be7a-003581644c67-kube-api-access-9v8r8\") pod \"payload-processing-e2e-ait-55140dc4-8b8949786-lshn2\" (UID: \"e66f6275-5b9e-4426-be7a-003581644c67\") " pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:16.291150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.291136 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66f6275-5b9e-4426-be7a-003581644c67-plugins-config-volume\") pod \"payload-processing-e2e-ait-55140dc4-8b8949786-lshn2\" (UID: \"e66f6275-5b9e-4426-be7a-003581644c67\") " pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:16.291888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.291868 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66f6275-5b9e-4426-be7a-003581644c67-plugins-config-volume\") pod \"payload-processing-e2e-ait-55140dc4-8b8949786-lshn2\" (UID: \"e66f6275-5b9e-4426-be7a-003581644c67\") " pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:16.358512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.353989 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v8r8\" (UniqueName: \"kubernetes.io/projected/e66f6275-5b9e-4426-be7a-003581644c67-kube-api-access-9v8r8\") pod \"payload-processing-e2e-ait-55140dc4-8b8949786-lshn2\" (UID: \"e66f6275-5b9e-4426-be7a-003581644c67\") " pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:16.410852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.410559 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:16.745374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:16.745352 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2"] Oct 30 06:06:16.747098 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:16.747066 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66f6275_5b9e_4426_be7a_003581644c67.slice/crio-90ab4cd485e7b45a48f50aaa51251b5a3a34b914f266999c720f1d5c0ff480bc WatchSource:0}: Error finding container 90ab4cd485e7b45a48f50aaa51251b5a3a34b914f266999c720f1d5c0ff480bc: Status 404 returned error can't find the container with id 90ab4cd485e7b45a48f50aaa51251b5a3a34b914f266999c720f1d5c0ff480bc Oct 30 06:06:17.023299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:17.023195 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" event={"ID":"e66f6275-5b9e-4426-be7a-003581644c67","Type":"ContainerStarted","Data":"16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92"} Oct 30 06:06:17.023299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:17.023274 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" event={"ID":"e66f6275-5b9e-4426-be7a-003581644c67","Type":"ContainerStarted","Data":"90ab4cd485e7b45a48f50aaa51251b5a3a34b914f266999c720f1d5c0ff480bc"} Oct 30 06:06:17.023836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:17.023648 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:17.078947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:17.076648 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" podStartSLOduration=1.076630998 podStartE2EDuration="1.076630998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:17.075767116 +0000 UTC m=+791.071724128" watchObservedRunningTime="2026-10-30 06:06:17.076630998 +0000 UTC m=+791.072588011" Oct 30 06:06:17.922369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:17.922328 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r"] Oct 30 06:06:17.926830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:17.926809 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:17.946570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:17.946547 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3627d700-openshift-default-dockercfg-dx6hx\"" Oct 30 06:06:17.988513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:17.988480 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r"] Oct 30 06:06:18.019184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.019149 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istiod-ca-cert\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.019404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.019206 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5fl\" (UniqueName: \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-kube-api-access-sp5fl\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.019404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.019272 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-data\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.019404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.019327 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-socket\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.019404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.019354 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-envoy\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.019404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.019375 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-token\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.019664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.019404 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-credential-socket\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.019664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.019432 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-certs\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.019664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.019463 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-podinfo\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.133824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.133784 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-data\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.135498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.134521 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-socket\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.135498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.134860 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-envoy\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.135498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.134891 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-data\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.135498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.134898 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-token\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.135498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.134984 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-credential-socket\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.135498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.135020 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-certs\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.135498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.135053 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-podinfo\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.135498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.135101 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-socket\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.135498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.135175 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istiod-ca-cert\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.135498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.135257 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5fl\" (UniqueName: \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-kube-api-access-sp5fl\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.135498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.135400 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-credential-socket\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.136219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.135550 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-certs\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.136686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.136660 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istiod-ca-cert\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.137832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.137801 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-envoy\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.138498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.138463 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-podinfo\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.194748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.194667 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-token\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.201173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.201150 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5fl\" (UniqueName: \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-kube-api-access-sp5fl\") pod \"e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.244028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.243985 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:18.647844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.647803 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r"] Oct 30 06:06:18.682200 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:18.681469 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47498426_9b5b_4d2c_bbe4_9856d6a7430d.slice/crio-cff4187d87ac99b31424092cd2d2f0d5a280bb2fb5db64657d0eca2981580497 WatchSource:0}: Error finding container cff4187d87ac99b31424092cd2d2f0d5a280bb2fb5db64657d0eca2981580497: Status 404 returned error can't find the container with id cff4187d87ac99b31424092cd2d2f0d5a280bb2fb5db64657d0eca2981580497 Oct 30 06:06:18.684629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.684588 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:18.684738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.684678 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:18.684738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:18.684716 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:19.088833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:19.088750 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" event={"ID":"47498426-9b5b-4d2c-bbe4-9856d6a7430d","Type":"ContainerStarted","Data":"e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170"} Oct 30 06:06:19.088833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:19.088808 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" event={"ID":"47498426-9b5b-4d2c-bbe4-9856d6a7430d","Type":"ContainerStarted","Data":"cff4187d87ac99b31424092cd2d2f0d5a280bb2fb5db64657d0eca2981580497"} Oct 30 06:06:19.244568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:19.244533 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:19.250099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:19.250077 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:19.309992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:19.309907 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" podStartSLOduration=2.309884334 podStartE2EDuration="2.309884334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:19.176527635 +0000 UTC m=+793.172484645" watchObservedRunningTime="2026-10-30 06:06:19.309884334 +0000 UTC m=+793.305841345" Oct 30 06:06:20.095355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:20.095292 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:20.096367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:20.096346 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:22.799005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:22.798972 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz"] Oct 30 06:06:22.803113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:22.803090 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:06:22.824580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:22.824548 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz"] Oct 30 06:06:22.834022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:22.833981 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3627d700\"" Oct 30 06:06:22.834022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:22.833986 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3627d700-dockercfg-vdjpv\"" Oct 30 06:06:22.951864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:22.951815 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz\" (UID: \"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:06:22.952224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:22.952000 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966hb\" (UniqueName: \"kubernetes.io/projected/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-kube-api-access-966hb\") pod \"payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz\" (UID: \"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:06:23.053398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:23.053293 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966hb\" (UniqueName: \"kubernetes.io/projected/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-kube-api-access-966hb\") pod \"payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz\" (UID: \"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:06:23.053586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:23.053470 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz\" (UID: \"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:06:23.054192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:23.054171 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz\" (UID: \"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:06:23.078152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:23.078123 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-966hb\" (UniqueName: \"kubernetes.io/projected/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-kube-api-access-966hb\") pod \"payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz\" (UID: \"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:06:23.119686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:23.119647 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:06:23.435857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:23.435832 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz"] Oct 30 06:06:23.437967 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:23.437938 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215fe8a3_5ff2_4197_90e9_4eba3b09dcf9.slice/crio-9ce7d7c64e9bc241ec5e58323718c1cef43c106d5b55c7cb5548f537a5ed2f47 WatchSource:0}: Error finding container 9ce7d7c64e9bc241ec5e58323718c1cef43c106d5b55c7cb5548f537a5ed2f47: Status 404 returned error can't find the container with id 9ce7d7c64e9bc241ec5e58323718c1cef43c106d5b55c7cb5548f537a5ed2f47 Oct 30 06:06:24.131290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:24.131256 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" event={"ID":"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9","Type":"ContainerStarted","Data":"bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b"} Oct 30 06:06:24.131736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:24.131328 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" event={"ID":"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9","Type":"ContainerStarted","Data":"9ce7d7c64e9bc241ec5e58323718c1cef43c106d5b55c7cb5548f537a5ed2f47"} Oct 30 06:06:24.131736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:24.131358 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:06:24.176716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:24.176652 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" podStartSLOduration=2.176632905 podStartE2EDuration="2.176632905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:24.174530674 +0000 UTC m=+798.170487694" watchObservedRunningTime="2026-10-30 06:06:24.176632905 +0000 UTC m=+798.172589916" Oct 30 06:06:24.355479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:24.355436 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:06:25.301955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.301924 5001 patch_prober.go:29] interesting pod/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:53060->10.132.0.54:15021: read: connection reset by peer" start-of-body= Oct 30 06:06:25.302393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.301983 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:53060->10.132.0.54:15021: read: connection reset by peer" Oct 30 06:06:25.798002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.797968 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p"] Oct 30 06:06:25.804601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.804576 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:25.820178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.820140 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:06:25.833001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.832969 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:06:25.833173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.833008 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:06:25.833173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.833068 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:06:25.835838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.835813 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p"] Oct 30 06:06:25.893943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.893907 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgb52\" (UniqueName: \"kubernetes.io/projected/a70876d7-f52e-4ece-a00a-0634c486bc8f-kube-api-access-xgb52\") pod \"maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:25.894147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.893963 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:25.894147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.894107 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:25.995187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.995148 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:25.995353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.995289 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgb52\" (UniqueName: \"kubernetes.io/projected/a70876d7-f52e-4ece-a00a-0634c486bc8f-kube-api-access-xgb52\") pod \"maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:25.995415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.995368 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:25.995941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.995905 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:25.997914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:25.997896 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:26.019836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:26.019803 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgb52\" (UniqueName: \"kubernetes.io/projected/a70876d7-f52e-4ece-a00a-0634c486bc8f-kube-api-access-xgb52\") pod \"maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:26.121464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:26.121429 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:26.149623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:26.149592 5001 generic.go:356] "Generic (PLEG): container finished" podID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerID="3df2ec8899b98a23f8154d6b1c8b6f80166b94706f8377c139e17073c8b358aa" exitCode=0 Oct 30 06:06:26.149798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:26.149666 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" event={"ID":"83901a1f-9a22-4dbd-899d-a44f2e916a65","Type":"ContainerDied","Data":"3df2ec8899b98a23f8154d6b1c8b6f80166b94706f8377c139e17073c8b358aa"} Oct 30 06:06:26.150161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:26.150141 5001 scope.go:122] "RemoveContainer" containerID="3df2ec8899b98a23f8154d6b1c8b6f80166b94706f8377c139e17073c8b358aa" Oct 30 06:06:26.151762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:26.151731 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:26.151950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:26.151822 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:26.151950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:26.151881 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:26.470429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:26.470397 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p"] Oct 30 06:06:26.470843 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:26.470649 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70876d7_f52e_4ece_a00a_0634c486bc8f.slice/crio-512af4600f10e01f2578e2a45613998c39e8adf772236f3ca9535e08d1be3816 WatchSource:0}: Error finding container 512af4600f10e01f2578e2a45613998c39e8adf772236f3ca9535e08d1be3816: Status 404 returned error can't find the container with id 512af4600f10e01f2578e2a45613998c39e8adf772236f3ca9535e08d1be3816 Oct 30 06:06:27.026675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:27.026646 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:27.026874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:27.026723 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:27.026874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:27.026765 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:27.195869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:27.195828 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" event={"ID":"83901a1f-9a22-4dbd-899d-a44f2e916a65","Type":"ContainerStarted","Data":"f70cf5d8d63b188c1c9bbee6c834aad54bfd087c6417d00cdcf5cdb99345a4c3"} Oct 30 06:06:27.197572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:27.197543 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" event={"ID":"a70876d7-f52e-4ece-a00a-0634c486bc8f","Type":"ContainerStarted","Data":"512af4600f10e01f2578e2a45613998c39e8adf772236f3ca9535e08d1be3816"} Oct 30 06:06:27.715152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:27.715116 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:27.715574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:27.715272 5001 patch_prober.go:29] interesting pod/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 06:06:27.715574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:27.715356 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 06:06:28.055561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:28.055475 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:28.714700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:28.714663 5001 patch_prober.go:29] interesting pod/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 06:06:28.715419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:28.715011 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 06:06:29.715036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:29.715001 5001 patch_prober.go:29] interesting pod/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 06:06:29.715233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:29.715058 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 06:06:30.715626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:30.715593 5001 patch_prober.go:29] interesting pod/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 06:06:30.716015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:30.715649 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 06:06:31.225377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.225339 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" event={"ID":"a70876d7-f52e-4ece-a00a-0634c486bc8f","Type":"ContainerStarted","Data":"3d987c743ca510a4bada3ba201df18890881be47a6eb53b6771cd6903a8d67bc"} Oct 30 06:06:31.301033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.300981 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" podStartSLOduration=2.533551718 podStartE2EDuration="6.300966204s" totalImagesPullingTime="3.767414486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:31.298138517 +0000 UTC m=+805.294095526" watchObservedRunningTime="2026-10-30 06:06:31.300966204 +0000 UTC m=+805.296923214" Oct 30 06:06:31.428780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.428750 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw"] Oct 30 06:06:31.432942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.432925 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:31.481380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.480281 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:31.481380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.480438 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvw5\" (UniqueName: \"kubernetes.io/projected/ccc52abc-de09-45bf-a88e-37fc12ddb82e-kube-api-access-pzvw5\") pod \"maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:31.481380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.480538 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:31.487467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.487435 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw"] Oct 30 06:06:31.582145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.582101 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:31.582359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.582167 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvw5\" (UniqueName: \"kubernetes.io/projected/ccc52abc-de09-45bf-a88e-37fc12ddb82e-kube-api-access-pzvw5\") pod \"maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:31.582359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.582237 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:31.583057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.583032 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:31.584944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.584920 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:31.612383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.612354 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvw5\" (UniqueName: \"kubernetes.io/projected/ccc52abc-de09-45bf-a88e-37fc12ddb82e-kube-api-access-pzvw5\") pod \"maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:31.715587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.715550 5001 patch_prober.go:29] interesting pod/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 06:06:31.715759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.715622 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 06:06:31.746351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:31.746246 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:32.083915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:32.083886 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw"] Oct 30 06:06:32.085943 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:32.085914 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc52abc_de09_45bf_a88e_37fc12ddb82e.slice/crio-3055fb8e8db7886a0ab25eadcf3247b53d97f296d895f4b7c5f4925d19dc5bf2 WatchSource:0}: Error finding container 3055fb8e8db7886a0ab25eadcf3247b53d97f296d895f4b7c5f4925d19dc5bf2: Status 404 returned error can't find the container with id 3055fb8e8db7886a0ab25eadcf3247b53d97f296d895f4b7c5f4925d19dc5bf2 Oct 30 06:06:32.233560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:32.233522 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" event={"ID":"ccc52abc-de09-45bf-a88e-37fc12ddb82e","Type":"ContainerStarted","Data":"3055fb8e8db7886a0ab25eadcf3247b53d97f296d895f4b7c5f4925d19dc5bf2"} Oct 30 06:06:32.235399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:32.235371 5001 generic.go:356] "Generic (PLEG): container finished" podID="a70876d7-f52e-4ece-a00a-0634c486bc8f" containerID="3d987c743ca510a4bada3ba201df18890881be47a6eb53b6771cd6903a8d67bc" exitCode=0 Oct 30 06:06:32.235531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:32.235429 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" event={"ID":"a70876d7-f52e-4ece-a00a-0634c486bc8f","Type":"ContainerDied","Data":"3d987c743ca510a4bada3ba201df18890881be47a6eb53b6771cd6903a8d67bc"} Oct 30 06:06:32.731111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:32.731017 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:32.732109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:32.732071 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:32.733166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:32.733136 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:33.243856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.243825 5001 generic.go:356] "Generic (PLEG): container finished" podID="ccc52abc-de09-45bf-a88e-37fc12ddb82e" containerID="e7c4ba65db23976950e691fb3f3cc9bfb5ebd92beff0f1ccd542c2c6e08207f7" exitCode=0 Oct 30 06:06:33.244051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.243909 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" event={"ID":"ccc52abc-de09-45bf-a88e-37fc12ddb82e","Type":"ContainerDied","Data":"e7c4ba65db23976950e691fb3f3cc9bfb5ebd92beff0f1ccd542c2c6e08207f7"} Oct 30 06:06:33.397114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.397093 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:33.505647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.505614 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-openshift-ca\") pod \"a70876d7-f52e-4ece-a00a-0634c486bc8f\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " Oct 30 06:06:33.505835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.505729 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-xks-ca\") pod \"a70876d7-f52e-4ece-a00a-0634c486bc8f\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " Oct 30 06:06:33.505885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.505833 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70876d7-f52e-4ece-a00a-0634c486bc8f-kube-api-access-xgb52\" (UniqueName: \"kubernetes.io/projected/a70876d7-f52e-4ece-a00a-0634c486bc8f-kube-api-access-xgb52\") pod \"a70876d7-f52e-4ece-a00a-0634c486bc8f\" (UID: \"a70876d7-f52e-4ece-a00a-0634c486bc8f\") " Oct 30 06:06:33.506051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.506023 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-openshift-ca" pod "a70876d7-f52e-4ece-a00a-0634c486bc8f" (UID: "a70876d7-f52e-4ece-a00a-0634c486bc8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:33.506227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.506207 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:33.508235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.508206 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70876d7-f52e-4ece-a00a-0634c486bc8f-kube-api-access-xgb52" pod "a70876d7-f52e-4ece-a00a-0634c486bc8f" (UID: "a70876d7-f52e-4ece-a00a-0634c486bc8f"). InnerVolumeSpecName "kube-api-access-xgb52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:33.508365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.508338 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-xks-ca" pod "a70876d7-f52e-4ece-a00a-0634c486bc8f" (UID: "a70876d7-f52e-4ece-a00a-0634c486bc8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:33.607519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.607494 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgb52\" (UniqueName: \"kubernetes.io/projected/a70876d7-f52e-4ece-a00a-0634c486bc8f-kube-api-access-xgb52\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:33.607668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:33.607523 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70876d7-f52e-4ece-a00a-0634c486bc8f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:34.257149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.257117 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" Oct 30 06:06:34.257618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.257145 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p" event={"ID":"a70876d7-f52e-4ece-a00a-0634c486bc8f","Type":"ContainerDied","Data":"512af4600f10e01f2578e2a45613998c39e8adf772236f3ca9535e08d1be3816"} Oct 30 06:06:34.257618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.257223 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="512af4600f10e01f2578e2a45613998c39e8adf772236f3ca9535e08d1be3816" Oct 30 06:06:34.451789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.451736 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:06:34.485416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.484151 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:34.624339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.621792 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc52abc-de09-45bf-a88e-37fc12ddb82e-kube-api-access-pzvw5\" (UniqueName: \"kubernetes.io/projected/ccc52abc-de09-45bf-a88e-37fc12ddb82e-kube-api-access-pzvw5\") pod \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " Oct 30 06:06:34.624339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.621853 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-xks-ca\") pod \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " Oct 30 06:06:34.624339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.621946 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-openshift-ca\") pod \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\" (UID: \"ccc52abc-de09-45bf-a88e-37fc12ddb82e\") " Oct 30 06:06:34.624339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.622613 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-openshift-ca" pod "ccc52abc-de09-45bf-a88e-37fc12ddb82e" (UID: "ccc52abc-de09-45bf-a88e-37fc12ddb82e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:34.630704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.630675 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc52abc-de09-45bf-a88e-37fc12ddb82e-kube-api-access-pzvw5" pod "ccc52abc-de09-45bf-a88e-37fc12ddb82e" (UID: "ccc52abc-de09-45bf-a88e-37fc12ddb82e"). InnerVolumeSpecName "kube-api-access-pzvw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:34.649638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.649610 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-xks-ca" pod "ccc52abc-de09-45bf-a88e-37fc12ddb82e" (UID: "ccc52abc-de09-45bf-a88e-37fc12ddb82e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:34.726630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.724335 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzvw5\" (UniqueName: \"kubernetes.io/projected/ccc52abc-de09-45bf-a88e-37fc12ddb82e-kube-api-access-pzvw5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:34.726630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.724379 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:34.726630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:34.724412 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc52abc-de09-45bf-a88e-37fc12ddb82e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:35.142976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:35.142948 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:06:35.265749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:35.265707 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" event={"ID":"ccc52abc-de09-45bf-a88e-37fc12ddb82e","Type":"ContainerDied","Data":"3055fb8e8db7886a0ab25eadcf3247b53d97f296d895f4b7c5f4925d19dc5bf2"} Oct 30 06:06:35.265749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:35.265747 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw" Oct 30 06:06:35.266292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:35.265769 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3055fb8e8db7886a0ab25eadcf3247b53d97f296d895f4b7c5f4925d19dc5bf2" Oct 30 06:06:36.247095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:36.247062 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v"] Oct 30 06:06:38.247646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.247603 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6"] Oct 30 06:06:38.248200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.248134 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ccc52abc-de09-45bf-a88e-37fc12ddb82e" containerName="revoke-keys" Oct 30 06:06:38.248200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.248147 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc52abc-de09-45bf-a88e-37fc12ddb82e" containerName="revoke-keys" Oct 30 06:06:38.248200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.248160 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a70876d7-f52e-4ece-a00a-0634c486bc8f" containerName="revoke-keys" Oct 30 06:06:38.248200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.248166 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70876d7-f52e-4ece-a00a-0634c486bc8f" containerName="revoke-keys" Oct 30 06:06:38.248447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.248255 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc52abc-de09-45bf-a88e-37fc12ddb82e" containerName="revoke-keys" Oct 30 06:06:38.248447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.248265 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70876d7-f52e-4ece-a00a-0634c486bc8f" containerName="revoke-keys" Oct 30 06:06:38.268634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.268602 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:38.282327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.275505 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6"] Oct 30 06:06:38.288869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.288840 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:06:38.298365 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:06:38.294495 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:06:38.301891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.301862 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:06:38.302218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.302198 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:06:38.302290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.302258 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:06:38.384591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.384552 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5l2d\" (UniqueName: \"kubernetes.io/projected/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-kube-api-access-b5l2d\") pod \"maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:38.384846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.384818 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:38.384952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.384888 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:38.487113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.486474 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:38.487519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.487484 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5l2d\" (UniqueName: \"kubernetes.io/projected/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-kube-api-access-b5l2d\") pod \"maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:38.487624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.487545 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:38.488142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.488119 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:38.489331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.489292 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:38.515495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.515427 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5l2d\" (UniqueName: \"kubernetes.io/projected/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-kube-api-access-b5l2d\") pod \"maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:38.602917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.602878 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:38.977327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:38.977261 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6"] Oct 30 06:06:38.981547 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:38.981523 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0481bbe6_ad0c_44af_93ba_64bcb51c1af0.slice/crio-77a4d5fd4e5cb83b010a35cfec6c0ebe86fb8c14bcacb9314497b9fe45415196 WatchSource:0}: Error finding container 77a4d5fd4e5cb83b010a35cfec6c0ebe86fb8c14bcacb9314497b9fe45415196: Status 404 returned error can't find the container with id 77a4d5fd4e5cb83b010a35cfec6c0ebe86fb8c14bcacb9314497b9fe45415196 Oct 30 06:06:39.305062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:39.305027 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" event={"ID":"0481bbe6-ad0c-44af-93ba-64bcb51c1af0","Type":"ContainerStarted","Data":"e16c706a4ee36097b034325fbe8d1e7c5c129fe3a0fa0439314566c60acf3916"} Oct 30 06:06:39.305498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:39.305081 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" event={"ID":"0481bbe6-ad0c-44af-93ba-64bcb51c1af0","Type":"ContainerStarted","Data":"77a4d5fd4e5cb83b010a35cfec6c0ebe86fb8c14bcacb9314497b9fe45415196"} Oct 30 06:06:39.346795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:39.346753 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" podStartSLOduration=1.346739302 podStartE2EDuration="1.346739302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:39.346366429 +0000 UTC m=+813.342323439" watchObservedRunningTime="2026-10-30 06:06:39.346739302 +0000 UTC m=+813.342696312" Oct 30 06:06:40.312277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:40.312185 5001 generic.go:356] "Generic (PLEG): container finished" podID="0481bbe6-ad0c-44af-93ba-64bcb51c1af0" containerID="e16c706a4ee36097b034325fbe8d1e7c5c129fe3a0fa0439314566c60acf3916" exitCode=0 Oct 30 06:06:40.312277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:40.312270 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" event={"ID":"0481bbe6-ad0c-44af-93ba-64bcb51c1af0","Type":"ContainerDied","Data":"e16c706a4ee36097b034325fbe8d1e7c5c129fe3a0fa0439314566c60acf3916"} Oct 30 06:06:41.247670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.247614 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" podUID="812f3d18-f834-4704-96cc-e830ffcca304" containerName="payload-pre-processing" containerID="cri-o://d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f" gracePeriod=25 Oct 30 06:06:41.477658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.477625 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9"] Oct 30 06:06:41.497892 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.497833 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:41.544820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.543533 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:41.634257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.634225 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-xks-ca\") pod \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " Oct 30 06:06:41.634464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.634327 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-openshift-ca\") pod \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " Oct 30 06:06:41.634464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.634402 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-kube-api-access-b5l2d\" (UniqueName: \"kubernetes.io/projected/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-kube-api-access-b5l2d\") pod \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\" (UID: \"0481bbe6-ad0c-44af-93ba-64bcb51c1af0\") " Oct 30 06:06:41.634779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.634751 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-openshift-ca" pod "0481bbe6-ad0c-44af-93ba-64bcb51c1af0" (UID: "0481bbe6-ad0c-44af-93ba-64bcb51c1af0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:41.635001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.634958 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:41.636604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.636576 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-xks-ca" pod "0481bbe6-ad0c-44af-93ba-64bcb51c1af0" (UID: "0481bbe6-ad0c-44af-93ba-64bcb51c1af0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:41.636753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.636730 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-kube-api-access-b5l2d" pod "0481bbe6-ad0c-44af-93ba-64bcb51c1af0" (UID: "0481bbe6-ad0c-44af-93ba-64bcb51c1af0"). InnerVolumeSpecName "kube-api-access-b5l2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:41.735904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.735868 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812f3d18-f834-4704-96cc-e830ffcca304-kube-api-access-qt9m4\" (UniqueName: \"kubernetes.io/projected/812f3d18-f834-4704-96cc-e830ffcca304-kube-api-access-qt9m4\") pod \"812f3d18-f834-4704-96cc-e830ffcca304\" (UID: \"812f3d18-f834-4704-96cc-e830ffcca304\") " Oct 30 06:06:41.736094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.735958 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/812f3d18-f834-4704-96cc-e830ffcca304-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/812f3d18-f834-4704-96cc-e830ffcca304-plugins-config-volume\") pod \"812f3d18-f834-4704-96cc-e830ffcca304\" (UID: \"812f3d18-f834-4704-96cc-e830ffcca304\") " Oct 30 06:06:41.736389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.736365 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/812f3d18-f834-4704-96cc-e830ffcca304-plugins-config-volume" pod "812f3d18-f834-4704-96cc-e830ffcca304" (UID: "812f3d18-f834-4704-96cc-e830ffcca304"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:41.736529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.736513 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:41.736595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.736535 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5l2d\" (UniqueName: \"kubernetes.io/projected/0481bbe6-ad0c-44af-93ba-64bcb51c1af0-kube-api-access-b5l2d\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:41.736595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.736549 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/812f3d18-f834-4704-96cc-e830ffcca304-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:41.738230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.738210 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812f3d18-f834-4704-96cc-e830ffcca304-kube-api-access-qt9m4" pod "812f3d18-f834-4704-96cc-e830ffcca304" (UID: "812f3d18-f834-4704-96cc-e830ffcca304"). InnerVolumeSpecName "kube-api-access-qt9m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:41.837209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:41.837137 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt9m4\" (UniqueName: \"kubernetes.io/projected/812f3d18-f834-4704-96cc-e830ffcca304-kube-api-access-qt9m4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:42.265639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.265605 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt"] Oct 30 06:06:42.330558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.330520 5001 generic.go:356] "Generic (PLEG): container finished" podID="812f3d18-f834-4704-96cc-e830ffcca304" containerID="d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f" exitCode=0 Oct 30 06:06:42.330758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.330652 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" Oct 30 06:06:42.330758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.330688 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" event={"ID":"812f3d18-f834-4704-96cc-e830ffcca304","Type":"ContainerDied","Data":"d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f"} Oct 30 06:06:42.330758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.330738 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v" event={"ID":"812f3d18-f834-4704-96cc-e830ffcca304","Type":"ContainerDied","Data":"c8f0342e8fee184625c72530412bad0fcddc0852715c0100b0b2b8e3ba927ade"} Oct 30 06:06:42.330917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.330772 5001 scope.go:122] "RemoveContainer" containerID="d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f" Oct 30 06:06:42.334191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.334110 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" event={"ID":"0481bbe6-ad0c-44af-93ba-64bcb51c1af0","Type":"ContainerDied","Data":"77a4d5fd4e5cb83b010a35cfec6c0ebe86fb8c14bcacb9314497b9fe45415196"} Oct 30 06:06:42.334191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.334168 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77a4d5fd4e5cb83b010a35cfec6c0ebe86fb8c14bcacb9314497b9fe45415196" Oct 30 06:06:42.334416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.334210 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6" Oct 30 06:06:42.350339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.350267 5001 scope.go:122] "RemoveContainer" containerID="d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f" Oct 30 06:06:42.350857 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:06:42.350807 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f\": container with ID starting with d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f not found: ID does not exist" containerID="d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f" Oct 30 06:06:42.350968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.350849 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f"} err="failed to get container status \"d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f\": rpc error: code = NotFound desc = could not find container \"d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f\": container with ID starting with d64533e9df57f6999650fee3c7febbfa9bf303f9572b625b38fbde454f5b2c7f not found: ID does not exist" Oct 30 06:06:42.410663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.410577 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v"] Oct 30 06:06:42.422153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:42.422118 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-798193ae-787bd8875-pml7v"] Oct 30 06:06:44.116546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:44.116513 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="812f3d18-f834-4704-96cc-e830ffcca304" path="/var/lib/kubelet/pods/812f3d18-f834-4704-96cc-e830ffcca304/volumes" Oct 30 06:06:45.354166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:45.354137 5001 generic.go:356] "Generic (PLEG): container finished" podID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerID="e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170" exitCode=0 Oct 30 06:06:45.354616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:45.354207 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" event={"ID":"47498426-9b5b-4d2c-bbe4-9856d6a7430d","Type":"ContainerDied","Data":"e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170"} Oct 30 06:06:45.354658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:45.354640 5001 scope.go:122] "RemoveContainer" containerID="e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170" Oct 30 06:06:45.355913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:45.355886 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:45.355997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:45.355958 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:45.356036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:45.356019 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:46.384901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:46.384869 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" event={"ID":"47498426-9b5b-4d2c-bbe4-9856d6a7430d","Type":"ContainerStarted","Data":"9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71"} Oct 30 06:06:46.479328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:46.478386 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" podUID="83764313-b8cd-4158-b83c-5436bfc4018f" containerName="payload-pre-processing" containerID="cri-o://4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d" gracePeriod=25 Oct 30 06:06:46.840801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:46.840777 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:46.913432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:46.912272 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83764313-b8cd-4158-b83c-5436bfc4018f-kube-api-access-8qgdj\" (UniqueName: \"kubernetes.io/projected/83764313-b8cd-4158-b83c-5436bfc4018f-kube-api-access-8qgdj\") pod \"83764313-b8cd-4158-b83c-5436bfc4018f\" (UID: \"83764313-b8cd-4158-b83c-5436bfc4018f\") " Oct 30 06:06:46.913432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:46.912388 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83764313-b8cd-4158-b83c-5436bfc4018f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83764313-b8cd-4158-b83c-5436bfc4018f-plugins-config-volume\") pod \"83764313-b8cd-4158-b83c-5436bfc4018f\" (UID: \"83764313-b8cd-4158-b83c-5436bfc4018f\") " Oct 30 06:06:46.913432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:46.913165 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83764313-b8cd-4158-b83c-5436bfc4018f-plugins-config-volume" pod "83764313-b8cd-4158-b83c-5436bfc4018f" (UID: "83764313-b8cd-4158-b83c-5436bfc4018f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:46.923949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:46.923904 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83764313-b8cd-4158-b83c-5436bfc4018f-kube-api-access-8qgdj" pod "83764313-b8cd-4158-b83c-5436bfc4018f" (UID: "83764313-b8cd-4158-b83c-5436bfc4018f"). InnerVolumeSpecName "kube-api-access-8qgdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:47.011916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.011823 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p"] Oct 30 06:06:47.014268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.014240 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qgdj\" (UniqueName: \"kubernetes.io/projected/83764313-b8cd-4158-b83c-5436bfc4018f-kube-api-access-8qgdj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.014268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.014266 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83764313-b8cd-4158-b83c-5436bfc4018f-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.014468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.014409 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-798193ae-ffk5p"] Oct 30 06:06:47.238871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.238834 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v"] Oct 30 06:06:47.239115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.239087 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" containerID="cri-o://f70cf5d8d63b188c1c9bbee6c834aad54bfd087c6417d00cdcf5cdb99345a4c3" gracePeriod=30 Oct 30 06:06:47.245278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.245067 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:47.245819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.245794 5001 patch_prober.go:29] interesting pod/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 06:06:47.245918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.245846 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 06:06:47.278251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.277354 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" podUID="8396fa23-d5d9-4ee9-a6f9-b3cd4484b105" containerName="payload-pre-processing" containerID="cri-o://d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a" gracePeriod=25 Oct 30 06:06:47.391543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.391515 5001 generic.go:356] "Generic (PLEG): container finished" podID="83764313-b8cd-4158-b83c-5436bfc4018f" containerID="4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d" exitCode=0 Oct 30 06:06:47.392009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.391570 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" Oct 30 06:06:47.392009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.391598 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" event={"ID":"83764313-b8cd-4158-b83c-5436bfc4018f","Type":"ContainerDied","Data":"4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d"} Oct 30 06:06:47.392009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.391668 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9" event={"ID":"83764313-b8cd-4158-b83c-5436bfc4018f","Type":"ContainerDied","Data":"33fb4fa86d795d21f6c594c4234c2ddd2dff8d9ae32c44b72e972447d4c47138"} Oct 30 06:06:47.392009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.391703 5001 scope.go:122] "RemoveContainer" containerID="4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d" Oct 30 06:06:47.401498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.401432 5001 scope.go:122] "RemoveContainer" containerID="4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d" Oct 30 06:06:47.401724 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:06:47.401706 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d\": container with ID starting with 4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d not found: ID does not exist" containerID="4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d" Oct 30 06:06:47.401769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.401733 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d"} err="failed to get container status \"4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d\": rpc error: code = NotFound desc = could not find container \"4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d\": container with ID starting with 4a0f9e32ce9b50df7dbe4002f24c33ba2d833341a43f9efbcede5323de76681d not found: ID does not exist" Oct 30 06:06:47.447255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.447226 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9"] Oct 30 06:06:47.482456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.477317 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c45a2ff8-5dd6fc9fc9-jsnh9"] Oct 30 06:06:47.712054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.712027 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:47.731133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.731106 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-kube-api-access-6z6t2\" (UniqueName: \"kubernetes.io/projected/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-kube-api-access-6z6t2\") pod \"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105\" (UID: \"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105\") " Oct 30 06:06:47.731329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.731270 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-plugins-config-volume\") pod \"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105\" (UID: \"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105\") " Oct 30 06:06:47.732028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.731986 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-plugins-config-volume" pod "8396fa23-d5d9-4ee9-a6f9-b3cd4484b105" (UID: "8396fa23-d5d9-4ee9-a6f9-b3cd4484b105"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:47.741935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.741879 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-kube-api-access-6z6t2" pod "8396fa23-d5d9-4ee9-a6f9-b3cd4484b105" (UID: "8396fa23-d5d9-4ee9-a6f9-b3cd4484b105"). InnerVolumeSpecName "kube-api-access-6z6t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:47.742392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.742366 5001 patch_prober.go:29] interesting pod/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:06:47.742546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.742418 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:06:47.833122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.833084 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.833594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:47.833572 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z6t2\" (UniqueName: \"kubernetes.io/projected/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105-kube-api-access-6z6t2\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:48.117141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.117112 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83764313-b8cd-4158-b83c-5436bfc4018f" path="/var/lib/kubelet/pods/83764313-b8cd-4158-b83c-5436bfc4018f/volumes" Oct 30 06:06:48.117567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.117550 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70876d7-f52e-4ece-a00a-0634c486bc8f" path="/var/lib/kubelet/pods/a70876d7-f52e-4ece-a00a-0634c486bc8f/volumes" Oct 30 06:06:48.128273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.128247 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw"] Oct 30 06:06:48.138467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.138438 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9c94fbc2-bwjjw"] Oct 30 06:06:48.244393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.244361 5001 patch_prober.go:29] interesting pod/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 06:06:48.244565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.244413 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 06:06:48.400337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.400212 5001 generic.go:356] "Generic (PLEG): container finished" podID="8396fa23-d5d9-4ee9-a6f9-b3cd4484b105" containerID="d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a" exitCode=0 Oct 30 06:06:48.400337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.400267 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" Oct 30 06:06:48.400952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.400354 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" event={"ID":"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105","Type":"ContainerDied","Data":"d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a"} Oct 30 06:06:48.400952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.400414 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt" event={"ID":"8396fa23-d5d9-4ee9-a6f9-b3cd4484b105","Type":"ContainerDied","Data":"05747b6cdeffbc647c030794c30bc1e1d1a30cb432de2eb7fd4a3dcb36741976"} Oct 30 06:06:48.400952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.400448 5001 scope.go:122] "RemoveContainer" containerID="d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a" Oct 30 06:06:48.409736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.409715 5001 scope.go:122] "RemoveContainer" containerID="d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a" Oct 30 06:06:48.410067 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:06:48.410046 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a\": container with ID starting with d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a not found: ID does not exist" containerID="d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a" Oct 30 06:06:48.410173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.410074 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a"} err="failed to get container status \"d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a\": rpc error: code = NotFound desc = could not find container \"d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a\": container with ID starting with d2148b87a9bc3877f8081bc028dbb64cd6240d826fef5988ae8aec4d4fa5e23a not found: ID does not exist" Oct 30 06:06:48.476523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.472212 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt"] Oct 30 06:06:48.481690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.481650 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9c94fbc2-7d8d55f76c-nnkrt"] Oct 30 06:06:48.768397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.765854 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2"] Oct 30 06:06:48.996342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.995536 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g"] Oct 30 06:06:48.996342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996328 5001 cpu_manager.go:414] "containerMap: removing container" podUID="83764313-b8cd-4158-b83c-5436bfc4018f" containerName="payload-pre-processing" Oct 30 06:06:48.996342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996347 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83764313-b8cd-4158-b83c-5436bfc4018f" containerName="payload-pre-processing" Oct 30 06:06:48.996635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996365 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8396fa23-d5d9-4ee9-a6f9-b3cd4484b105" containerName="payload-pre-processing" Oct 30 06:06:48.996635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996372 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8396fa23-d5d9-4ee9-a6f9-b3cd4484b105" containerName="payload-pre-processing" Oct 30 06:06:48.996635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996397 5001 cpu_manager.go:414] "containerMap: removing container" podUID="812f3d18-f834-4704-96cc-e830ffcca304" containerName="payload-pre-processing" Oct 30 06:06:48.996635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996404 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="812f3d18-f834-4704-96cc-e830ffcca304" containerName="payload-pre-processing" Oct 30 06:06:48.996635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996440 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0481bbe6-ad0c-44af-93ba-64bcb51c1af0" containerName="revoke-keys" Oct 30 06:06:48.996635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996447 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0481bbe6-ad0c-44af-93ba-64bcb51c1af0" containerName="revoke-keys" Oct 30 06:06:48.996635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996541 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="812f3d18-f834-4704-96cc-e830ffcca304" containerName="payload-pre-processing" Oct 30 06:06:48.996635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996555 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0481bbe6-ad0c-44af-93ba-64bcb51c1af0" containerName="revoke-keys" Oct 30 06:06:48.996635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996566 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="83764313-b8cd-4158-b83c-5436bfc4018f" containerName="payload-pre-processing" Oct 30 06:06:48.996635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:48.996574 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8396fa23-d5d9-4ee9-a6f9-b3cd4484b105" containerName="payload-pre-processing" Oct 30 06:06:49.002360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.001544 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.021209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.021091 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-aaf467-openshift-default-dockercfg-txgqd\"" Oct 30 06:06:49.024317 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.024276 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g"] Oct 30 06:06:49.106431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.106397 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.106431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.106431 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.106658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.106450 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.106658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.106513 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.106658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.106611 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.106791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.106677 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4wxw\" (UniqueName: \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-kube-api-access-b4wxw\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.106791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.106707 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.106791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.106778 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.106898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.106813 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.207964 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wxw\" (UniqueName: \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-kube-api-access-b4wxw\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208021 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208179 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208258 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208371 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208402 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208430 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208465 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208473 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208527 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208562 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208705 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.208954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.208931 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.209550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.209529 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.211590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.211571 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.211777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.211760 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.232263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.232230 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wxw\" (UniqueName: \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-kube-api-access-b4wxw\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.232659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.232637 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.244764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.244740 5001 patch_prober.go:29] interesting pod/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 06:06:49.244861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.244781 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 06:06:49.318622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.318528 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:49.684501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.684459 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g"] Oct 30 06:06:49.685342 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:49.685294 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1d0f14_43a9_4609_a7c7_de84d0c47758.slice/crio-4acc464bfdd4718fff74df3fbc7417b333dc58e2f28e4d384755aa8d06b9582c WatchSource:0}: Error finding container 4acc464bfdd4718fff74df3fbc7417b333dc58e2f28e4d384755aa8d06b9582c: Status 404 returned error can't find the container with id 4acc464bfdd4718fff74df3fbc7417b333dc58e2f28e4d384755aa8d06b9582c Oct 30 06:06:49.687825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.687796 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:49.687935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.687877 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:49.688004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:49.687933 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:50.097411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:50.096507 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:50.148869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:50.148837 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8396fa23-d5d9-4ee9-a6f9-b3cd4484b105" path="/var/lib/kubelet/pods/8396fa23-d5d9-4ee9-a6f9-b3cd4484b105/volumes" Oct 30 06:06:50.149744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:50.149720 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc52abc-de09-45bf-a88e-37fc12ddb82e" path="/var/lib/kubelet/pods/ccc52abc-de09-45bf-a88e-37fc12ddb82e/volumes" Oct 30 06:06:50.245460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:50.245398 5001 patch_prober.go:29] interesting pod/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 06:06:50.245632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:50.245462 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 06:06:50.427456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:50.427419 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" event={"ID":"9d1d0f14-43a9-4609-a7c7-de84d0c47758","Type":"ContainerStarted","Data":"df8c3f90fe3de3afc2bd82234e12a9a7bd1c48b0185a54e845b3940f62dfb7f4"} Oct 30 06:06:50.427646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:50.427471 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" event={"ID":"9d1d0f14-43a9-4609-a7c7-de84d0c47758","Type":"ContainerStarted","Data":"4acc464bfdd4718fff74df3fbc7417b333dc58e2f28e4d384755aa8d06b9582c"} Oct 30 06:06:50.489232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:50.489170 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" podStartSLOduration=2.4891565079999998 podStartE2EDuration="2.489156508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:50.487532569 +0000 UTC m=+824.483489580" watchObservedRunningTime="2026-10-30 06:06:50.489156508 +0000 UTC m=+824.485113586" Oct 30 06:06:51.031432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:51.031393 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:06:51.245520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:51.244729 5001 patch_prober.go:29] interesting pod/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 06:06:51.245520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:51.244792 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 06:06:51.319166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:51.319084 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:51.324172 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:51.324141 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:51.435843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:51.435808 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:51.436979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:51.436961 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:06:52.316845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.316818 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:52.318023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.318002 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:06:52.448080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.448050 5001 generic.go:356] "Generic (PLEG): container finished" podID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerID="f70cf5d8d63b188c1c9bbee6c834aad54bfd087c6417d00cdcf5cdb99345a4c3" exitCode=0 Oct 30 06:06:52.448863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.448825 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" event={"ID":"83901a1f-9a22-4dbd-899d-a44f2e916a65","Type":"ContainerDied","Data":"f70cf5d8d63b188c1c9bbee6c834aad54bfd087c6417d00cdcf5cdb99345a4c3"} Oct 30 06:06:52.449056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.448888 5001 scope.go:122] "RemoveContainer" containerID="3df2ec8899b98a23f8154d6b1c8b6f80166b94706f8377c139e17073c8b358aa" Oct 30 06:06:52.515570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.515542 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:52.551627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.551555 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-podinfo\") pod \"83901a1f-9a22-4dbd-899d-a44f2e916a65\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " Oct 30 06:06:52.551749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.551649 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83901a1f-9a22-4dbd-899d-a44f2e916a65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83901a1f-9a22-4dbd-899d-a44f2e916a65-istiod-ca-cert\") pod \"83901a1f-9a22-4dbd-899d-a44f2e916a65\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " Oct 30 06:06:52.551749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.551693 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-envoy\") pod \"83901a1f-9a22-4dbd-899d-a44f2e916a65\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " Oct 30 06:06:52.551749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.551716 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-socket\") pod \"83901a1f-9a22-4dbd-899d-a44f2e916a65\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " Oct 30 06:06:52.551885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.551792 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-token\" (UniqueName: \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-token\") pod \"83901a1f-9a22-4dbd-899d-a44f2e916a65\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " Oct 30 06:06:52.551885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.551836 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-credential-socket\") pod \"83901a1f-9a22-4dbd-899d-a44f2e916a65\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " Oct 30 06:06:52.551885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.551875 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-data\") pod \"83901a1f-9a22-4dbd-899d-a44f2e916a65\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " Oct 30 06:06:52.552072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.551914 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-certs\") pod \"83901a1f-9a22-4dbd-899d-a44f2e916a65\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " Oct 30 06:06:52.552072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.551974 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-kube-api-access-z88lb\" (UniqueName: \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-kube-api-access-z88lb\") pod \"83901a1f-9a22-4dbd-899d-a44f2e916a65\" (UID: \"83901a1f-9a22-4dbd-899d-a44f2e916a65\") " Oct 30 06:06:52.552072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.552045 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-socket" pod "83901a1f-9a22-4dbd-899d-a44f2e916a65" (UID: "83901a1f-9a22-4dbd-899d-a44f2e916a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:52.552228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.552084 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83901a1f-9a22-4dbd-899d-a44f2e916a65-istiod-ca-cert" pod "83901a1f-9a22-4dbd-899d-a44f2e916a65" (UID: "83901a1f-9a22-4dbd-899d-a44f2e916a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:52.552288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.552239 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-data" pod "83901a1f-9a22-4dbd-899d-a44f2e916a65" (UID: "83901a1f-9a22-4dbd-899d-a44f2e916a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:52.552511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.552459 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-certs" pod "83901a1f-9a22-4dbd-899d-a44f2e916a65" (UID: "83901a1f-9a22-4dbd-899d-a44f2e916a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:52.552746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.552704 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-credential-socket" pod "83901a1f-9a22-4dbd-899d-a44f2e916a65" (UID: "83901a1f-9a22-4dbd-899d-a44f2e916a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:52.553372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.553137 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.553372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.553164 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.553372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.553179 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.553372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.553192 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83901a1f-9a22-4dbd-899d-a44f2e916a65-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.553372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.553204 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.554505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.554477 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-token" pod "83901a1f-9a22-4dbd-899d-a44f2e916a65" (UID: "83901a1f-9a22-4dbd-899d-a44f2e916a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:52.554730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.554712 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-envoy" pod "83901a1f-9a22-4dbd-899d-a44f2e916a65" (UID: "83901a1f-9a22-4dbd-899d-a44f2e916a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:52.555188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.555163 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-kube-api-access-z88lb" pod "83901a1f-9a22-4dbd-899d-a44f2e916a65" (UID: "83901a1f-9a22-4dbd-899d-a44f2e916a65"). InnerVolumeSpecName "kube-api-access-z88lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:52.555443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.555418 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-podinfo" pod "83901a1f-9a22-4dbd-899d-a44f2e916a65" (UID: "83901a1f-9a22-4dbd-899d-a44f2e916a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:06:52.653785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.653746 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.653785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.653777 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z88lb\" (UniqueName: \"kubernetes.io/projected/83901a1f-9a22-4dbd-899d-a44f2e916a65-kube-api-access-z88lb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.653785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.653791 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.654013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.653803 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83901a1f-9a22-4dbd-899d-a44f2e916a65-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.658402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.658373 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6"] Oct 30 06:06:52.660621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:52.660598 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55140dc4-s2dc6"] Oct 30 06:06:53.460294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:53.460265 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" Oct 30 06:06:53.460968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:53.460388 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v" event={"ID":"83901a1f-9a22-4dbd-899d-a44f2e916a65","Type":"ContainerDied","Data":"bed72328a22686ba6abe558006443371dc1f0906ee57a80ed6b15de8e2026fc3"} Oct 30 06:06:53.461111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:53.460997 5001 scope.go:122] "RemoveContainer" containerID="f70cf5d8d63b188c1c9bbee6c834aad54bfd087c6417d00cdcf5cdb99345a4c3" Oct 30 06:06:53.516852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:53.516815 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v"] Oct 30 06:06:53.529961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:53.529923 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-798193ae-gw-openshift-default-58c6597d6f-l254v"] Oct 30 06:06:53.767272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:53.767163 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" podUID="e66f6275-5b9e-4426-be7a-003581644c67" containerName="payload-processing" containerID="cri-o://16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92" gracePeriod=25 Oct 30 06:06:54.016836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.016812 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:54.077270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.077188 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66f6275-5b9e-4426-be7a-003581644c67-kube-api-access-9v8r8\" (UniqueName: \"kubernetes.io/projected/e66f6275-5b9e-4426-be7a-003581644c67-kube-api-access-9v8r8\") pod \"e66f6275-5b9e-4426-be7a-003581644c67\" (UID: \"e66f6275-5b9e-4426-be7a-003581644c67\") " Oct 30 06:06:54.077461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.077321 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e66f6275-5b9e-4426-be7a-003581644c67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66f6275-5b9e-4426-be7a-003581644c67-plugins-config-volume\") pod \"e66f6275-5b9e-4426-be7a-003581644c67\" (UID: \"e66f6275-5b9e-4426-be7a-003581644c67\") " Oct 30 06:06:54.077711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.077684 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66f6275-5b9e-4426-be7a-003581644c67-plugins-config-volume" pod "e66f6275-5b9e-4426-be7a-003581644c67" (UID: "e66f6275-5b9e-4426-be7a-003581644c67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:54.079654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.079625 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66f6275-5b9e-4426-be7a-003581644c67-kube-api-access-9v8r8" pod "e66f6275-5b9e-4426-be7a-003581644c67" (UID: "e66f6275-5b9e-4426-be7a-003581644c67"). InnerVolumeSpecName "kube-api-access-9v8r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:54.124075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.124034 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0481bbe6-ad0c-44af-93ba-64bcb51c1af0" path="/var/lib/kubelet/pods/0481bbe6-ad0c-44af-93ba-64bcb51c1af0/volumes" Oct 30 06:06:54.124589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.124566 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" path="/var/lib/kubelet/pods/83901a1f-9a22-4dbd-899d-a44f2e916a65/volumes" Oct 30 06:06:54.178880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.178846 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v8r8\" (UniqueName: \"kubernetes.io/projected/e66f6275-5b9e-4426-be7a-003581644c67-kube-api-access-9v8r8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:54.178880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.178881 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66f6275-5b9e-4426-be7a-003581644c67-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:06:54.470496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.470459 5001 generic.go:356] "Generic (PLEG): container finished" podID="e66f6275-5b9e-4426-be7a-003581644c67" containerID="16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92" exitCode=0 Oct 30 06:06:54.470975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.470503 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" event={"ID":"e66f6275-5b9e-4426-be7a-003581644c67","Type":"ContainerDied","Data":"16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92"} Oct 30 06:06:54.470975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.470545 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" Oct 30 06:06:54.470975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.470560 5001 scope.go:122] "RemoveContainer" containerID="16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92" Oct 30 06:06:54.470975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.470549 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2" event={"ID":"e66f6275-5b9e-4426-be7a-003581644c67","Type":"ContainerDied","Data":"90ab4cd485e7b45a48f50aaa51251b5a3a34b914f266999c720f1d5c0ff480bc"} Oct 30 06:06:54.482087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.482062 5001 scope.go:122] "RemoveContainer" containerID="16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92" Oct 30 06:06:54.482416 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:06:54.482386 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92\": container with ID starting with 16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92 not found: ID does not exist" containerID="16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92" Oct 30 06:06:54.482513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.482429 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92"} err="failed to get container status \"16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92\": rpc error: code = NotFound desc = could not find container \"16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92\": container with ID starting with 16d8b824931134998606c65eb068346a113ad2cda0a406465abd32ab46a73c92 not found: ID does not exist" Oct 30 06:06:54.531083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.531050 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2"] Oct 30 06:06:54.544477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.544449 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55140dc4-8b8949786-lshn2"] Oct 30 06:06:54.713589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.713556 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g"] Oct 30 06:06:54.714155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.714136 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e66f6275-5b9e-4426-be7a-003581644c67" containerName="payload-processing" Oct 30 06:06:54.714215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.714158 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66f6275-5b9e-4426-be7a-003581644c67" containerName="payload-processing" Oct 30 06:06:54.714215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.714172 5001 cpu_manager.go:414] "containerMap: removing container" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" Oct 30 06:06:54.714215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.714177 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" Oct 30 06:06:54.714215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.714189 5001 cpu_manager.go:414] "containerMap: removing container" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" Oct 30 06:06:54.714215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.714198 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" Oct 30 06:06:54.714419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.714345 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" Oct 30 06:06:54.714419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.714359 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="83901a1f-9a22-4dbd-899d-a44f2e916a65" containerName="istio-proxy" Oct 30 06:06:54.714419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.714379 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66f6275-5b9e-4426-be7a-003581644c67" containerName="payload-processing" Oct 30 06:06:54.719179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.719163 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 06:06:54.737669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.737641 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-8d6349\"" Oct 30 06:06:54.738101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.738076 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-8d6349-dockercfg-5cg6v\"" Oct 30 06:06:54.738343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.738293 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g"] Oct 30 06:06:54.785706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.785677 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc55x\" (UniqueName: \"kubernetes.io/projected/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-kube-api-access-jc55x\") pod \"payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g\" (UID: \"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 06:06:54.785884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.785746 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g\" (UID: \"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 06:06:54.898365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.895124 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc55x\" (UniqueName: \"kubernetes.io/projected/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-kube-api-access-jc55x\") pod \"payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g\" (UID: \"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 06:06:54.898365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.895262 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g\" (UID: \"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 06:06:54.898365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.896232 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g\" (UID: \"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 06:06:54.938141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:54.938075 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc55x\" (UniqueName: \"kubernetes.io/projected/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-kube-api-access-jc55x\") pod \"payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g\" (UID: \"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 06:06:55.032674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:55.032569 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 06:06:55.335940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:55.335767 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g"] Oct 30 06:06:55.338742 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:55.338714 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd74b66_5969_4ae8_bd39_57cb48fbd1e2.slice/crio-6896a7747b01170fe29bc1a5079dc21ca3413f3d1ddca0e62e332dccac12b831 WatchSource:0}: Error finding container 6896a7747b01170fe29bc1a5079dc21ca3413f3d1ddca0e62e332dccac12b831: Status 404 returned error can't find the container with id 6896a7747b01170fe29bc1a5079dc21ca3413f3d1ddca0e62e332dccac12b831 Oct 30 06:06:55.479518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:55.479481 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" event={"ID":"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2","Type":"ContainerStarted","Data":"075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8"} Oct 30 06:06:55.479876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:55.479527 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" event={"ID":"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2","Type":"ContainerStarted","Data":"6896a7747b01170fe29bc1a5079dc21ca3413f3d1ddca0e62e332dccac12b831"} Oct 30 06:06:55.479876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:55.479559 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 06:06:55.523374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:55.523300 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" podStartSLOduration=1.5232854329999999 podStartE2EDuration="1.523285433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:55.522036998 +0000 UTC m=+829.517994007" watchObservedRunningTime="2026-10-30 06:06:55.523285433 +0000 UTC m=+829.519242440" Oct 30 06:06:56.118006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.117976 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66f6275-5b9e-4426-be7a-003581644c67" path="/var/lib/kubelet/pods/e66f6275-5b9e-4426-be7a-003581644c67/volumes" Oct 30 06:06:56.376094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.376012 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh"] Oct 30 06:06:56.381802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.381775 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 06:06:56.399553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.399520 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh"] Oct 30 06:06:56.471387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.470640 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-aaf467\"" Oct 30 06:06:56.471387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.470698 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-aaf467-dockercfg-mzv85\"" Oct 30 06:06:56.517329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.517067 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5trmt\" (UniqueName: \"kubernetes.io/projected/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-kube-api-access-5trmt\") pod \"payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh\" (UID: \"0f4b00a4-8f73-4fa9-9e96-7100bce648a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 06:06:56.517329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.517151 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh\" (UID: \"0f4b00a4-8f73-4fa9-9e96-7100bce648a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 06:06:56.618270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.618232 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5trmt\" (UniqueName: \"kubernetes.io/projected/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-kube-api-access-5trmt\") pod \"payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh\" (UID: \"0f4b00a4-8f73-4fa9-9e96-7100bce648a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 06:06:56.618436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.618345 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh\" (UID: \"0f4b00a4-8f73-4fa9-9e96-7100bce648a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 06:06:56.619043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.619022 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh\" (UID: \"0f4b00a4-8f73-4fa9-9e96-7100bce648a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 06:06:56.643363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.643261 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5trmt\" (UniqueName: \"kubernetes.io/projected/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-kube-api-access-5trmt\") pod \"payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh\" (UID: \"0f4b00a4-8f73-4fa9-9e96-7100bce648a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 06:06:56.695703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:56.695674 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 06:06:57.023367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.019820 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh"] Oct 30 06:06:57.292142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.290879 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9"] Oct 30 06:06:57.296613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.296584 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:06:57.314733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.314702 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9"] Oct 30 06:06:57.315763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.315737 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-eaee88\"" Oct 30 06:06:57.316367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.316348 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-eaee88-dockercfg-r6k72\"" Oct 30 06:06:57.388363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.381203 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwqv7\" (UniqueName: \"kubernetes.io/projected/a315417d-889e-448b-b3fd-ebf54e93aac1-kube-api-access-kwqv7\") pod \"payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9\" (UID: \"a315417d-889e-448b-b3fd-ebf54e93aac1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:06:57.388363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.381325 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a315417d-889e-448b-b3fd-ebf54e93aac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9\" (UID: \"a315417d-889e-448b-b3fd-ebf54e93aac1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:06:57.405732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.405691 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd"] Oct 30 06:06:57.412975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.412950 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:06:57.413501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.413069 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd"] Oct 30 06:06:57.481930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.481900 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwqv7\" (UniqueName: \"kubernetes.io/projected/a315417d-889e-448b-b3fd-ebf54e93aac1-kube-api-access-kwqv7\") pod \"payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9\" (UID: \"a315417d-889e-448b-b3fd-ebf54e93aac1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:06:57.482128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.481967 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr2jw\" (UniqueName: \"kubernetes.io/projected/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-kube-api-access-wr2jw\") pod \"payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd\" (UID: \"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:06:57.482128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.482022 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a315417d-889e-448b-b3fd-ebf54e93aac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9\" (UID: \"a315417d-889e-448b-b3fd-ebf54e93aac1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:06:57.482128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.482056 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd\" (UID: \"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:06:57.483025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.482998 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a315417d-889e-448b-b3fd-ebf54e93aac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9\" (UID: \"a315417d-889e-448b-b3fd-ebf54e93aac1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:06:57.506481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.506446 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" event={"ID":"0f4b00a4-8f73-4fa9-9e96-7100bce648a8","Type":"ContainerStarted","Data":"1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2"} Oct 30 06:06:57.506629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.506499 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" event={"ID":"0f4b00a4-8f73-4fa9-9e96-7100bce648a8","Type":"ContainerStarted","Data":"d783d0fe7e07eb8e8a1c1423871ccf2d6daf226af5d40fffebb3d5fd120095ae"} Oct 30 06:06:57.506629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.506518 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwqv7\" (UniqueName: \"kubernetes.io/projected/a315417d-889e-448b-b3fd-ebf54e93aac1-kube-api-access-kwqv7\") pod \"payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9\" (UID: \"a315417d-889e-448b-b3fd-ebf54e93aac1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:06:57.506629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.506559 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 06:06:57.549178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.549091 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" podStartSLOduration=1.54907773 podStartE2EDuration="1.54907773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:57.547163198 +0000 UTC m=+831.543120232" watchObservedRunningTime="2026-10-30 06:06:57.54907773 +0000 UTC m=+831.545034756" Oct 30 06:06:57.582711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.582677 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2jw\" (UniqueName: \"kubernetes.io/projected/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-kube-api-access-wr2jw\") pod \"payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd\" (UID: \"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:06:57.582867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.582747 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd\" (UID: \"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:06:57.583534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.583509 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd\" (UID: \"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:06:57.607012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.606983 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2jw\" (UniqueName: \"kubernetes.io/projected/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-kube-api-access-wr2jw\") pod \"payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd\" (UID: \"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:06:57.610884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.610852 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:06:57.732388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.732352 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:06:57.936460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:57.936436 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9"] Oct 30 06:06:57.938674 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:06:57.938625 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda315417d_889e_448b_b3fd_ebf54e93aac1.slice/crio-7336f0d4f31ccea545e69745ae6eaee10f7213943ec27293f54344c5c130dc86 WatchSource:0}: Error finding container 7336f0d4f31ccea545e69745ae6eaee10f7213943ec27293f54344c5c130dc86: Status 404 returned error can't find the container with id 7336f0d4f31ccea545e69745ae6eaee10f7213943ec27293f54344c5c130dc86 Oct 30 06:06:58.042924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:58.042899 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd"] Oct 30 06:06:58.514868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:58.514831 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" event={"ID":"a315417d-889e-448b-b3fd-ebf54e93aac1","Type":"ContainerStarted","Data":"36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189"} Oct 30 06:06:58.515053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:58.514893 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" event={"ID":"a315417d-889e-448b-b3fd-ebf54e93aac1","Type":"ContainerStarted","Data":"7336f0d4f31ccea545e69745ae6eaee10f7213943ec27293f54344c5c130dc86"} Oct 30 06:06:58.515053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:58.514921 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:06:58.517102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:58.517075 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" event={"ID":"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb","Type":"ContainerStarted","Data":"e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94"} Oct 30 06:06:58.517219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:58.517129 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" event={"ID":"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb","Type":"ContainerStarted","Data":"62f0c72b6c77426910e6c3323aab7e7361e6872b18570419366052435e021e74"} Oct 30 06:06:58.517219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:58.517159 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:06:58.569939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:58.569894 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" podStartSLOduration=1.569880556 podStartE2EDuration="1.569880556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:58.568951128 +0000 UTC m=+832.564908138" watchObservedRunningTime="2026-10-30 06:06:58.569880556 +0000 UTC m=+832.565837565" Oct 30 06:06:58.620680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:58.620617 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" podStartSLOduration=1.620597507 podStartE2EDuration="1.620597507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:58.616610751 +0000 UTC m=+832.612567776" watchObservedRunningTime="2026-10-30 06:06:58.620597507 +0000 UTC m=+832.616554517" Oct 30 06:06:59.185907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:06:59.185874 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz"] Oct 30 06:07:01.548330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:01.547373 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:07:02.008744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.008709 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg"] Oct 30 06:07:02.014235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.014208 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:07:02.026635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.026607 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg\" (UID: \"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:07:02.026800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.026774 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qqr4\" (UniqueName: \"kubernetes.io/projected/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-kube-api-access-6qqr4\") pod \"payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg\" (UID: \"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:07:02.042058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.042027 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a519e9-dockercfg-mq22m\"" Oct 30 06:07:02.045199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.045172 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg"] Oct 30 06:07:02.055138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.055106 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a519e9\"" Oct 30 06:07:02.157611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.157574 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qqr4\" (UniqueName: \"kubernetes.io/projected/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-kube-api-access-6qqr4\") pod \"payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg\" (UID: \"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:07:02.158290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.158262 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg\" (UID: \"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:07:02.162237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.162186 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg\" (UID: \"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:07:02.215110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.215075 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qqr4\" (UniqueName: \"kubernetes.io/projected/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-kube-api-access-6qqr4\") pod \"payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg\" (UID: \"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:07:02.329066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.328972 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:07:02.647136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:02.647097 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg"] Oct 30 06:07:02.648902 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:07:02.648878 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1e5e0a_f7a7_4a53_968a_3e8404fdce87.slice/crio-b1eb089f5a322bc1d9150bd2e770aa99cbc6ff7d7231432690b956b39f4a29ef WatchSource:0}: Error finding container b1eb089f5a322bc1d9150bd2e770aa99cbc6ff7d7231432690b956b39f4a29ef: Status 404 returned error can't find the container with id b1eb089f5a322bc1d9150bd2e770aa99cbc6ff7d7231432690b956b39f4a29ef Oct 30 06:07:03.588427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:03.588380 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" event={"ID":"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87","Type":"ContainerStarted","Data":"55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f"} Oct 30 06:07:03.588591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:03.588445 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" event={"ID":"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87","Type":"ContainerStarted","Data":"b1eb089f5a322bc1d9150bd2e770aa99cbc6ff7d7231432690b956b39f4a29ef"} Oct 30 06:07:03.588591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:03.588572 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:07:03.636689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:03.636618 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" podStartSLOduration=2.636600426 podStartE2EDuration="2.636600426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:03.635908392 +0000 UTC m=+837.631865404" watchObservedRunningTime="2026-10-30 06:07:03.636600426 +0000 UTC m=+837.632557436" Oct 30 06:07:04.187089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.187038 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" podUID="215fe8a3-5ff2-4197-90e9-4eba3b09dcf9" containerName="payload-pre-processing" containerID="cri-o://bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b" gracePeriod=25 Oct 30 06:07:04.525096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.525069 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:07:04.605454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.605423 5001 generic.go:356] "Generic (PLEG): container finished" podID="215fe8a3-5ff2-4197-90e9-4eba3b09dcf9" containerID="bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b" exitCode=0 Oct 30 06:07:04.605616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.605484 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" Oct 30 06:07:04.605616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.605507 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" event={"ID":"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9","Type":"ContainerDied","Data":"bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b"} Oct 30 06:07:04.605616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.605573 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz" event={"ID":"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9","Type":"ContainerDied","Data":"9ce7d7c64e9bc241ec5e58323718c1cef43c106d5b55c7cb5548f537a5ed2f47"} Oct 30 06:07:04.605616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.605598 5001 scope.go:122] "RemoveContainer" containerID="bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b" Oct 30 06:07:04.615248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.615231 5001 scope.go:122] "RemoveContainer" containerID="bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b" Oct 30 06:07:04.615551 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:07:04.615533 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b\": container with ID starting with bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b not found: ID does not exist" containerID="bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b" Oct 30 06:07:04.615618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.615558 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b"} err="failed to get container status \"bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b\": rpc error: code = NotFound desc = could not find container \"bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b\": container with ID starting with bec632227127eed6850568f10508adbd3b69569b0850e0aad7fa3c500b45ee3b not found: ID does not exist" Oct 30 06:07:04.699227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.699193 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-kube-api-access-966hb\" (UniqueName: \"kubernetes.io/projected/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-kube-api-access-966hb\") pod \"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9\" (UID: \"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9\") " Oct 30 06:07:04.699423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.699299 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-plugins-config-volume\") pod \"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9\" (UID: \"215fe8a3-5ff2-4197-90e9-4eba3b09dcf9\") " Oct 30 06:07:04.700000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.699971 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-plugins-config-volume" pod "215fe8a3-5ff2-4197-90e9-4eba3b09dcf9" (UID: "215fe8a3-5ff2-4197-90e9-4eba3b09dcf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:04.702259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.702232 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-kube-api-access-966hb" pod "215fe8a3-5ff2-4197-90e9-4eba3b09dcf9" (UID: "215fe8a3-5ff2-4197-90e9-4eba3b09dcf9"). InnerVolumeSpecName "kube-api-access-966hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:04.800721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.800636 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-966hb\" (UniqueName: \"kubernetes.io/projected/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-kube-api-access-966hb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:07:04.800721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.800669 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:07:04.980522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.980476 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz"] Oct 30 06:07:04.980724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:04.980544 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3627d700-74849467dc-497gz"] Oct 30 06:07:05.835152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:05.835113 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r"] Oct 30 06:07:05.835559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:05.835387 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" containerID="cri-o://9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71" gracePeriod=30 Oct 30 06:07:06.117546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:06.117514 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="215fe8a3-5ff2-4197-90e9-4eba3b09dcf9" path="/var/lib/kubelet/pods/215fe8a3-5ff2-4197-90e9-4eba3b09dcf9/volumes" Oct 30 06:07:06.503204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:06.503121 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 06:07:07.320195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:07.320119 5001 patch_prober.go:29] interesting pod/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:07:07.320195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:07.320186 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:07:07.780329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:07.765408 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:07:08.273911 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:07:08.273869 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:07:08.521424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:08.521393 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 06:07:09.535074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:09.535040 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:07:09.535513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:09.535202 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:07:10.967900 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:07:10.967853 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47498426_9b5b_4d2c_bbe4_9856d6a7430d.slice/crio-9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:07:10.968559 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:07:10.967865 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47498426_9b5b_4d2c_bbe4_9856d6a7430d.slice/crio-9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47498426_9b5b_4d2c_bbe4_9856d6a7430d.slice/crio-conmon-9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:07:10.968655 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:07:10.967882 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47498426_9b5b_4d2c_bbe4_9856d6a7430d.slice/crio-cff4187d87ac99b31424092cd2d2f0d5a280bb2fb5db64657d0eca2981580497\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47498426_9b5b_4d2c_bbe4_9856d6a7430d.slice/crio-9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47498426_9b5b_4d2c_bbe4_9856d6a7430d.slice/crio-conmon-9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:07:11.120517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.120495 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:07:11.198123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198095 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-certs\") pod \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " Oct 30 06:07:11.198123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198132 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istiod-ca-cert\") pod \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " Oct 30 06:07:11.198409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198150 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-socket\") pod \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " Oct 30 06:07:11.198409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198217 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-token\" (UniqueName: \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-token\") pod \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " Oct 30 06:07:11.198409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198265 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-kube-api-access-sp5fl\" (UniqueName: \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-kube-api-access-sp5fl\") pod \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " Oct 30 06:07:11.198409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198331 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-podinfo\") pod \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " Oct 30 06:07:11.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198410 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-data\") pod \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " Oct 30 06:07:11.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198434 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-certs" pod "47498426-9b5b-4d2c-bbe4-9856d6a7430d" (UID: "47498426-9b5b-4d2c-bbe4-9856d6a7430d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:11.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198446 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-credential-socket\") pod \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " Oct 30 06:07:11.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198503 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-socket" pod "47498426-9b5b-4d2c-bbe4-9856d6a7430d" (UID: "47498426-9b5b-4d2c-bbe4-9856d6a7430d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:11.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198528 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-envoy\") pod \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\" (UID: \"47498426-9b5b-4d2c-bbe4-9856d6a7430d\") " Oct 30 06:07:11.198925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198905 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:07:11.198980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.198932 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:07:11.199065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.199039 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-credential-socket" pod "47498426-9b5b-4d2c-bbe4-9856d6a7430d" (UID: "47498426-9b5b-4d2c-bbe4-9856d6a7430d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:11.199128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.199109 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-data" pod "47498426-9b5b-4d2c-bbe4-9856d6a7430d" (UID: "47498426-9b5b-4d2c-bbe4-9856d6a7430d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:11.199438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.199402 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istiod-ca-cert" pod "47498426-9b5b-4d2c-bbe4-9856d6a7430d" (UID: "47498426-9b5b-4d2c-bbe4-9856d6a7430d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:11.200757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.200735 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-podinfo" pod "47498426-9b5b-4d2c-bbe4-9856d6a7430d" (UID: "47498426-9b5b-4d2c-bbe4-9856d6a7430d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:07:11.201234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.201210 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-kube-api-access-sp5fl" pod "47498426-9b5b-4d2c-bbe4-9856d6a7430d" (UID: "47498426-9b5b-4d2c-bbe4-9856d6a7430d"). InnerVolumeSpecName "kube-api-access-sp5fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:11.201449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.201431 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-envoy" pod "47498426-9b5b-4d2c-bbe4-9856d6a7430d" (UID: "47498426-9b5b-4d2c-bbe4-9856d6a7430d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:11.201776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.201762 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-token" pod "47498426-9b5b-4d2c-bbe4-9856d6a7430d" (UID: "47498426-9b5b-4d2c-bbe4-9856d6a7430d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:11.299549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.299513 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:07:11.299549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.299542 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:07:11.299549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.299554 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp5fl\" (UniqueName: \"kubernetes.io/projected/47498426-9b5b-4d2c-bbe4-9856d6a7430d-kube-api-access-sp5fl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:07:11.299767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.299570 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:07:11.299767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.299582 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:07:11.299767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.299592 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:07:11.299767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.299602 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47498426-9b5b-4d2c-bbe4-9856d6a7430d-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:07:11.657414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.657371 5001 generic.go:356] "Generic (PLEG): container finished" podID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerID="9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71" exitCode=0 Oct 30 06:07:11.658995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.657427 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" Oct 30 06:07:11.658995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.657459 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" event={"ID":"47498426-9b5b-4d2c-bbe4-9856d6a7430d","Type":"ContainerDied","Data":"9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71"} Oct 30 06:07:11.658995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.657529 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r" event={"ID":"47498426-9b5b-4d2c-bbe4-9856d6a7430d","Type":"ContainerDied","Data":"cff4187d87ac99b31424092cd2d2f0d5a280bb2fb5db64657d0eca2981580497"} Oct 30 06:07:11.658995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.657561 5001 scope.go:122] "RemoveContainer" containerID="9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71" Oct 30 06:07:11.671428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.671406 5001 scope.go:122] "RemoveContainer" containerID="e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170" Oct 30 06:07:11.681002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.680980 5001 scope.go:122] "RemoveContainer" containerID="9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71" Oct 30 06:07:11.681342 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:07:11.681317 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71\": container with ID starting with 9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71 not found: ID does not exist" containerID="9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71" Oct 30 06:07:11.681412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.681355 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71"} err="failed to get container status \"9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71\": rpc error: code = NotFound desc = could not find container \"9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71\": container with ID starting with 9967449343d8b93c1f2bc88e413fcfb8205f737c49e2534fed328eccabcaec71 not found: ID does not exist" Oct 30 06:07:11.681412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.681379 5001 scope.go:122] "RemoveContainer" containerID="e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170" Oct 30 06:07:11.681652 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:07:11.681632 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170\": container with ID starting with e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170 not found: ID does not exist" containerID="e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170" Oct 30 06:07:11.681706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.681658 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170"} err="failed to get container status \"e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170\": rpc error: code = NotFound desc = could not find container \"e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170\": container with ID starting with e9d85fc74aeb4433fb0dd159e1f7336733abf84b62a07e47d60bffe3e0ee7170 not found: ID does not exist" Oct 30 06:07:11.727202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.727164 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r"] Oct 30 06:07:11.739136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:11.739099 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3627d700-openshift-default-7fc86ddc7-vqz6r"] Oct 30 06:07:12.116692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.116660 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" path="/var/lib/kubelet/pods/47498426-9b5b-4d2c-bbe4-9856d6a7430d/volumes" Oct 30 06:07:12.589460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.589378 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w"] Oct 30 06:07:12.590388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.590352 5001 cpu_manager.go:414] "containerMap: removing container" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" Oct 30 06:07:12.590388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.590379 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" Oct 30 06:07:12.590549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.590413 5001 cpu_manager.go:414] "containerMap: removing container" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" Oct 30 06:07:12.590549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.590421 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" Oct 30 06:07:12.590549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.590454 5001 cpu_manager.go:414] "containerMap: removing container" podUID="215fe8a3-5ff2-4197-90e9-4eba3b09dcf9" containerName="payload-pre-processing" Oct 30 06:07:12.590549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.590461 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="215fe8a3-5ff2-4197-90e9-4eba3b09dcf9" containerName="payload-pre-processing" Oct 30 06:07:12.590691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.590580 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" Oct 30 06:07:12.590691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.590596 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="47498426-9b5b-4d2c-bbe4-9856d6a7430d" containerName="istio-proxy" Oct 30 06:07:12.590691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.590610 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="215fe8a3-5ff2-4197-90e9-4eba3b09dcf9" containerName="payload-pre-processing" Oct 30 06:07:12.593137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.593120 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 06:07:12.611149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.611124 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-8d44dd-dockercfg-fvw74\"" Oct 30 06:07:12.611795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.611768 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-8d44dd\"" Oct 30 06:07:12.615287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.614862 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w"] Oct 30 06:07:12.615488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.615461 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154d7c27-437c-4fde-b888-e93026217476-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w\" (UID: \"154d7c27-437c-4fde-b888-e93026217476\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 06:07:12.615567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.615529 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czrz9\" (UniqueName: \"kubernetes.io/projected/154d7c27-437c-4fde-b888-e93026217476-kube-api-access-czrz9\") pod \"payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w\" (UID: \"154d7c27-437c-4fde-b888-e93026217476\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 06:07:12.642906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.642878 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg"] Oct 30 06:07:12.645686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.645667 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 06:07:12.669790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.669510 5001 generic.go:356] "Generic (PLEG): container finished" podID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerID="df8c3f90fe3de3afc2bd82234e12a9a7bd1c48b0185a54e845b3940f62dfb7f4" exitCode=0 Oct 30 06:07:12.669790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.669581 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" event={"ID":"9d1d0f14-43a9-4609-a7c7-de84d0c47758","Type":"ContainerDied","Data":"df8c3f90fe3de3afc2bd82234e12a9a7bd1c48b0185a54e845b3940f62dfb7f4"} Oct 30 06:07:12.670018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.669998 5001 scope.go:122] "RemoveContainer" containerID="df8c3f90fe3de3afc2bd82234e12a9a7bd1c48b0185a54e845b3940f62dfb7f4" Oct 30 06:07:12.671671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.671637 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:12.671749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.671728 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:12.671815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.671775 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:12.678580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.674830 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg"] Oct 30 06:07:12.732330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.716898 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154d7c27-437c-4fde-b888-e93026217476-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w\" (UID: \"154d7c27-437c-4fde-b888-e93026217476\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 06:07:12.732330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.716974 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czrz9\" (UniqueName: \"kubernetes.io/projected/154d7c27-437c-4fde-b888-e93026217476-kube-api-access-czrz9\") pod \"payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w\" (UID: \"154d7c27-437c-4fde-b888-e93026217476\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 06:07:12.732330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.717013 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8lfn\" (UniqueName: \"kubernetes.io/projected/50081a44-3b27-4b9d-8718-3b52afeb8fc4-kube-api-access-l8lfn\") pod \"payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg\" (UID: \"50081a44-3b27-4b9d-8718-3b52afeb8fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 06:07:12.732330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.717047 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50081a44-3b27-4b9d-8718-3b52afeb8fc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg\" (UID: \"50081a44-3b27-4b9d-8718-3b52afeb8fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 06:07:12.732330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.718354 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154d7c27-437c-4fde-b888-e93026217476-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w\" (UID: \"154d7c27-437c-4fde-b888-e93026217476\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 06:07:12.752332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.751967 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czrz9\" (UniqueName: \"kubernetes.io/projected/154d7c27-437c-4fde-b888-e93026217476-kube-api-access-czrz9\") pod \"payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w\" (UID: \"154d7c27-437c-4fde-b888-e93026217476\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 06:07:12.819598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.818289 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8lfn\" (UniqueName: \"kubernetes.io/projected/50081a44-3b27-4b9d-8718-3b52afeb8fc4-kube-api-access-l8lfn\") pod \"payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg\" (UID: \"50081a44-3b27-4b9d-8718-3b52afeb8fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 06:07:12.819598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.818380 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50081a44-3b27-4b9d-8718-3b52afeb8fc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg\" (UID: \"50081a44-3b27-4b9d-8718-3b52afeb8fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 06:07:12.819598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.819296 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50081a44-3b27-4b9d-8718-3b52afeb8fc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg\" (UID: \"50081a44-3b27-4b9d-8718-3b52afeb8fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 06:07:12.846026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.845929 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8lfn\" (UniqueName: \"kubernetes.io/projected/50081a44-3b27-4b9d-8718-3b52afeb8fc4-kube-api-access-l8lfn\") pod \"payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg\" (UID: \"50081a44-3b27-4b9d-8718-3b52afeb8fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 06:07:12.908018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.907984 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 06:07:12.966127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:12.965922 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 06:07:13.101131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.099907 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:07:13.235171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.235148 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w"] Oct 30 06:07:13.416493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.416471 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg"] Oct 30 06:07:13.419720 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:07:13.419655 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50081a44_3b27_4b9d_8718_3b52afeb8fc4.slice/crio-a6ba3d5fcb944df0b389761687e957498a93f36384c0680d42cb0089532803d3 WatchSource:0}: Error finding container a6ba3d5fcb944df0b389761687e957498a93f36384c0680d42cb0089532803d3: Status 404 returned error can't find the container with id a6ba3d5fcb944df0b389761687e957498a93f36384c0680d42cb0089532803d3 Oct 30 06:07:13.687995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.686931 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" event={"ID":"9d1d0f14-43a9-4609-a7c7-de84d0c47758","Type":"ContainerStarted","Data":"32200783d1486cc072fe792cadf3ed115fc012ca07faa87b47b30c0c0569a3db"} Oct 30 06:07:13.698953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.698907 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" event={"ID":"50081a44-3b27-4b9d-8718-3b52afeb8fc4","Type":"ContainerStarted","Data":"6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c"} Oct 30 06:07:13.699324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.699275 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" event={"ID":"50081a44-3b27-4b9d-8718-3b52afeb8fc4","Type":"ContainerStarted","Data":"a6ba3d5fcb944df0b389761687e957498a93f36384c0680d42cb0089532803d3"} Oct 30 06:07:13.703773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.701291 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 06:07:13.703773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.703535 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" event={"ID":"154d7c27-437c-4fde-b888-e93026217476","Type":"ContainerStarted","Data":"c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a"} Oct 30 06:07:13.703773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.703562 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" event={"ID":"154d7c27-437c-4fde-b888-e93026217476","Type":"ContainerStarted","Data":"52eb82c3249c441befeba01e57762241719872ab36c16933b5f58d60154cbae0"} Oct 30 06:07:13.703773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.703588 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 06:07:13.844433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.844376 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" podStartSLOduration=1.844361446 podStartE2EDuration="1.844361446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:13.841764296 +0000 UTC m=+847.837721305" watchObservedRunningTime="2026-10-30 06:07:13.844361446 +0000 UTC m=+847.840318457" Oct 30 06:07:13.892691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:13.892643 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" podStartSLOduration=1.892628893 podStartE2EDuration="1.892628893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:13.891243087 +0000 UTC m=+847.887200098" watchObservedRunningTime="2026-10-30 06:07:13.892628893 +0000 UTC m=+847.888585902" Oct 30 06:07:14.319170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:14.319136 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:07:14.319658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:14.319284 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 06:07:14.319658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:14.319365 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 06:07:14.608600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:14.608575 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:07:15.321694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:15.321060 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 06:07:15.321694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:15.321133 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 06:07:16.319707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:16.319674 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 06:07:16.319891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:16.319726 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 06:07:17.319188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:17.319103 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 06:07:17.319188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:17.319168 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 06:07:18.327650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:18.327615 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 06:07:18.328140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:18.327665 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 06:07:19.323535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:19.323501 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:07:19.323967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:19.323945 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:07:19.324638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:19.324618 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:07:24.667331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:24.666354 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:07:24.715789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:24.715759 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 06:07:25.722485 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:25.722457 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 06:07:30.136948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:30.136903 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:07:38.278934 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:07:38.278898 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:07:38.536648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:38.536565 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:07:49.526056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:49.526020 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:07:50.501605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.501567 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5"] Oct 30 06:07:50.507753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.507727 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.540707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.540674 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"default-dockercfg-jbw59\"" Oct 30 06:07:50.541146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.540717 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"simulator-w1-8d44dd-kserve-self-signed-certs\"" Oct 30 06:07:50.543838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.543811 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"openshift-service-ca.crt\"" Oct 30 06:07:50.544077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.544059 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"kube-root-ca.crt\"" Oct 30 06:07:50.545550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.545524 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5"] Oct 30 06:07:50.603580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.603545 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-home\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.603772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.603587 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpxhq\" (UniqueName: \"kubernetes.io/projected/96382c2d-6131-412c-96fb-42a678998e1d-kube-api-access-hpxhq\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.603772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.603719 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-dshm\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.603772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.603756 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-tmp-dir\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.603948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.603783 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-model-cache\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.603948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.603894 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96382c2d-6131-412c-96fb-42a678998e1d-tls-certs\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.705053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.705017 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-home\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.705053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.705061 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpxhq\" (UniqueName: \"kubernetes.io/projected/96382c2d-6131-412c-96fb-42a678998e1d-kube-api-access-hpxhq\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.705321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.705123 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-dshm\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.705321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.705148 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-tmp-dir\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.705321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.705269 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-model-cache\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.705498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.705364 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96382c2d-6131-412c-96fb-42a678998e1d-tls-certs\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.705498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.705474 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-home\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.705612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.705580 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-tmp-dir\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.705739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.705718 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-model-cache\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.707664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.707642 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-dshm\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.707989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.707973 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96382c2d-6131-412c-96fb-42a678998e1d-tls-certs\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.746460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.746378 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpxhq\" (UniqueName: \"kubernetes.io/projected/96382c2d-6131-412c-96fb-42a678998e1d-kube-api-access-hpxhq\") pod \"simulator-w1-8d44dd-kserve-6f776df956-rt9z5\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:50.822530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:50.822444 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:07:51.132831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:51.132807 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5"] Oct 30 06:07:51.134883 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:07:51.134853 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96382c2d_6131_412c_96fb_42a678998e1d.slice/crio-96b4100baa361d9193a98195acf4271df1b93a9a7424124bacc7d9b0acde244f WatchSource:0}: Error finding container 96b4100baa361d9193a98195acf4271df1b93a9a7424124bacc7d9b0acde244f: Status 404 returned error can't find the container with id 96b4100baa361d9193a98195acf4271df1b93a9a7424124bacc7d9b0acde244f Oct 30 06:07:51.980005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:51.979966 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" event={"ID":"96382c2d-6131-412c-96fb-42a678998e1d","Type":"ContainerStarted","Data":"a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d"} Oct 30 06:07:51.980005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:51.980009 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" event={"ID":"96382c2d-6131-412c-96fb-42a678998e1d","Type":"ContainerStarted","Data":"96b4100baa361d9193a98195acf4271df1b93a9a7424124bacc7d9b0acde244f"} Oct 30 06:07:52.023858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:52.023805 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podStartSLOduration=2.023790253 podStartE2EDuration="2.023790253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:52.022422769 +0000 UTC m=+886.018379789" watchObservedRunningTime="2026-10-30 06:07:52.023790253 +0000 UTC m=+886.019747262" Oct 30 06:07:52.823601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:07:52.823569 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:08:00.823377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:00.823341 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:08:00.835958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:00.835935 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:08:01.050010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:01.049982 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:08:01.435403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:01.435372 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:08:06.098246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.098211 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr"] Oct 30 06:08:06.104104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.104083 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.121821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.121795 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr"] Oct 30 06:08:06.131859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.131796 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"openshift-service-ca.crt\"" Oct 30 06:08:06.131859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.131796 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"kube-root-ca.crt\"" Oct 30 06:08:06.132067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.132052 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"premium-w0-a519e9-kserve-self-signed-certs\"" Oct 30 06:08:06.133403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.133381 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"default-dockercfg-9mttb\"" Oct 30 06:08:06.209847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.209821 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:08:06.214421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.214397 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:08:06.303508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.303484 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a721547-d091-4208-b0bd-4264a62c37cd-tls-certs\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.303668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.303519 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-dshm\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.303668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.303547 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-home\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.303668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.303594 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-model-cache\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.303668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.303649 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvtzg\" (UniqueName: \"kubernetes.io/projected/6a721547-d091-4208-b0bd-4264a62c37cd-kube-api-access-lvtzg\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.303862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.303731 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-tmp-dir\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.407331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.406958 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-home\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.407331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.407014 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-model-cache\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.407331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.407061 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvtzg\" (UniqueName: \"kubernetes.io/projected/6a721547-d091-4208-b0bd-4264a62c37cd-kube-api-access-lvtzg\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.407331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.407118 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-tmp-dir\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.407331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.407241 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a721547-d091-4208-b0bd-4264a62c37cd-tls-certs\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.407331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.407279 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-dshm\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.408734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.408705 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-home\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.408902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.408738 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-model-cache\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.409017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.408959 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-tmp-dir\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.417339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.415816 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-dshm\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.418065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.418015 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a721547-d091-4208-b0bd-4264a62c37cd-tls-certs\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.433020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.432992 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvtzg\" (UniqueName: \"kubernetes.io/projected/6a721547-d091-4208-b0bd-4264a62c37cd-kube-api-access-lvtzg\") pod \"premium-w0-a519e9-kserve-69798b5c88-5lwzr\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:06.719211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:06.719116 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:07.058349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:07.054924 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr"] Oct 30 06:08:07.081177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:07.081149 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" event={"ID":"6a721547-d091-4208-b0bd-4264a62c37cd","Type":"ContainerStarted","Data":"1072ba8450863d4f9db27c62969dfefca0a98b0b92e3e6a1063783df7ec2cfe7"} Oct 30 06:08:08.090698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:08.090631 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" event={"ID":"6a721547-d091-4208-b0bd-4264a62c37cd","Type":"ContainerStarted","Data":"4d47e2e1dd35b3c43751118a4ba5511e48785c2dad5c9f797e1a0f6aa8f2b61f"} Oct 30 06:08:08.137270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:08.137213 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podStartSLOduration=2.137199523 podStartE2EDuration="2.137199523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:08.136134249 +0000 UTC m=+902.132091262" watchObservedRunningTime="2026-10-30 06:08:08.137199523 +0000 UTC m=+902.133156534" Oct 30 06:08:08.282934 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:08:08.282898 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:08:08.719486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:08.719452 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:11.999332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:11.999263 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:08:16.720234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:16.720202 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:16.732898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:16.732871 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:17.163499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:17.163471 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:08:18.863718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.863689 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6"] Oct 30 06:08:18.868187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.868158 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:18.889366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.889344 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8d6349-openshift-default-dockercfg-xs2qm\"" Oct 30 06:08:18.900253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.900229 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6"] Oct 30 06:08:18.941987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.941947 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:18.942171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.942002 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:18.942171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.942115 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:18.942171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.942145 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:18.942296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.942172 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:18.942296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.942211 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngtv9\" (UniqueName: \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-kube-api-access-ngtv9\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:18.942296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.942289 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:18.942440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.942392 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:18.942574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:18.942560 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.043397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.043365 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.043597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.043413 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.043597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.043437 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.043597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.043474 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.043597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.043508 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.043597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.043559 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.043874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.043644 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.043874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.043680 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.043874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.043725 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngtv9\" (UniqueName: \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-kube-api-access-ngtv9\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.043874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.043837 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.044054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.043895 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.044188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.044162 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.044261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.044236 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.044388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.044368 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.046101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.046082 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.046580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.046559 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.068118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.068082 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.068330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.068163 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngtv9\" (UniqueName: \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-kube-api-access-ngtv9\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.184149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.184069 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:19.525166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.525141 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6"] Oct 30 06:08:19.527450 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:08:19.527421 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3a428c_c55f_4ea6_b9e2_be1dce741eaf.slice/crio-a380fd09c6a37efbb94d276e642963c72dd77a146bf1aa43bf7fa2add8e8395c WatchSource:0}: Error finding container a380fd09c6a37efbb94d276e642963c72dd77a146bf1aa43bf7fa2add8e8395c: Status 404 returned error can't find the container with id a380fd09c6a37efbb94d276e642963c72dd77a146bf1aa43bf7fa2add8e8395c Oct 30 06:08:19.529852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.529819 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:19.529950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.529896 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:19.529950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:19.529922 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:20.181333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:20.179610 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" event={"ID":"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf","Type":"ContainerStarted","Data":"61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a"} Oct 30 06:08:20.181333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:20.179679 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" event={"ID":"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf","Type":"ContainerStarted","Data":"a380fd09c6a37efbb94d276e642963c72dd77a146bf1aa43bf7fa2add8e8395c"} Oct 30 06:08:20.184836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:20.184811 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:20.186471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:20.186450 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:08:20.186567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:20.186492 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" podUID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 06:08:20.241925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:20.241870 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" podStartSLOduration=2.2418525 podStartE2EDuration="2.2418525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:20.227576253 +0000 UTC m=+914.223533263" watchObservedRunningTime="2026-10-30 06:08:20.2418525 +0000 UTC m=+914.237809510" Oct 30 06:08:21.184814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:21.184782 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:08:21.185459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:21.185416 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" podUID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 06:08:22.184868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:22.184822 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:08:22.185391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:22.184892 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" podUID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 06:08:23.185702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:23.185655 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:08:23.186193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:23.185739 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" podUID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 06:08:24.187474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:24.187410 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:08:24.187993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:24.187476 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" podUID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 06:08:25.187330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:25.184924 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:08:25.187330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:25.185000 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" podUID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 06:08:26.202996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:26.202959 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:26.203566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:26.203548 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:26.204698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:26.204674 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:08:33.627486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:33.627441 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:08:38.274334 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:08:38.271446 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:08:43.306346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.306291 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq"] Oct 30 06:08:43.310809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.310789 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:43.329631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.329606 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq"] Oct 30 06:08:43.339394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.339369 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:08:43.339593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.339575 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:08:43.339663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.339522 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:08:43.339663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.339476 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:08:43.379869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.379839 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b44eb62-lvcwq\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:43.380048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.379903 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b44eb62-lvcwq\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:43.380091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.380058 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhtt\" (UniqueName: \"kubernetes.io/projected/39d88a5f-de29-4751-bfb9-947f381498c4-kube-api-access-qmhtt\") pod \"maas-api-revoke-sub-4b44eb62-lvcwq\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:43.480887 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.480850 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b44eb62-lvcwq\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:43.481055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.481009 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhtt\" (UniqueName: \"kubernetes.io/projected/39d88a5f-de29-4751-bfb9-947f381498c4-kube-api-access-qmhtt\") pod \"maas-api-revoke-sub-4b44eb62-lvcwq\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:43.481101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.481074 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b44eb62-lvcwq\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:43.481783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.481755 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b44eb62-lvcwq\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:43.483395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.483372 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b44eb62-lvcwq\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:43.505653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.505620 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhtt\" (UniqueName: \"kubernetes.io/projected/39d88a5f-de29-4751-bfb9-947f381498c4-kube-api-access-qmhtt\") pod \"maas-api-revoke-sub-4b44eb62-lvcwq\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:43.632387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.632352 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:43.949520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:43.949492 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq"] Oct 30 06:08:43.952015 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:08:43.951977 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d88a5f_de29_4751_bfb9_947f381498c4.slice/crio-e501caa93c2f78d3f42a7aeb44096de8486ea94eb73adaa6d56f684fb539a705 WatchSource:0}: Error finding container e501caa93c2f78d3f42a7aeb44096de8486ea94eb73adaa6d56f684fb539a705: Status 404 returned error can't find the container with id e501caa93c2f78d3f42a7aeb44096de8486ea94eb73adaa6d56f684fb539a705 Oct 30 06:08:44.358115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:44.358083 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" event={"ID":"39d88a5f-de29-4751-bfb9-947f381498c4","Type":"ContainerStarted","Data":"c14e3cbed58eb7f574e65eff280648ffc7cffeb55abe776043a3f8f320824754"} Oct 30 06:08:44.358629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:44.358132 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" event={"ID":"39d88a5f-de29-4751-bfb9-947f381498c4","Type":"ContainerStarted","Data":"e501caa93c2f78d3f42a7aeb44096de8486ea94eb73adaa6d56f684fb539a705"} Oct 30 06:08:44.403513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:44.403458 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" podStartSLOduration=1.403445016 podStartE2EDuration="1.403445016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:44.402985943 +0000 UTC m=+938.398942953" watchObservedRunningTime="2026-10-30 06:08:44.403445016 +0000 UTC m=+938.399402039" Oct 30 06:08:45.365785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:45.365753 5001 generic.go:356] "Generic (PLEG): container finished" podID="39d88a5f-de29-4751-bfb9-947f381498c4" containerID="c14e3cbed58eb7f574e65eff280648ffc7cffeb55abe776043a3f8f320824754" exitCode=0 Oct 30 06:08:45.366233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:45.365809 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" event={"ID":"39d88a5f-de29-4751-bfb9-947f381498c4","Type":"ContainerDied","Data":"c14e3cbed58eb7f574e65eff280648ffc7cffeb55abe776043a3f8f320824754"} Oct 30 06:08:46.556862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:46.556837 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:46.611965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:46.611931 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d88a5f-de29-4751-bfb9-947f381498c4-kube-api-access-qmhtt\" (UniqueName: \"kubernetes.io/projected/39d88a5f-de29-4751-bfb9-947f381498c4-kube-api-access-qmhtt\") pod \"39d88a5f-de29-4751-bfb9-947f381498c4\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " Oct 30 06:08:46.612156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:46.612058 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-xks-ca\") pod \"39d88a5f-de29-4751-bfb9-947f381498c4\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " Oct 30 06:08:46.612156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:46.612109 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-openshift-ca\") pod \"39d88a5f-de29-4751-bfb9-947f381498c4\" (UID: \"39d88a5f-de29-4751-bfb9-947f381498c4\") " Oct 30 06:08:46.612538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:46.612515 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-openshift-ca" pod "39d88a5f-de29-4751-bfb9-947f381498c4" (UID: "39d88a5f-de29-4751-bfb9-947f381498c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:46.614768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:46.614616 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-xks-ca" pod "39d88a5f-de29-4751-bfb9-947f381498c4" (UID: "39d88a5f-de29-4751-bfb9-947f381498c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:46.615217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:46.615151 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d88a5f-de29-4751-bfb9-947f381498c4-kube-api-access-qmhtt" pod "39d88a5f-de29-4751-bfb9-947f381498c4" (UID: "39d88a5f-de29-4751-bfb9-947f381498c4"). InnerVolumeSpecName "kube-api-access-qmhtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:46.714057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:46.714018 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmhtt\" (UniqueName: \"kubernetes.io/projected/39d88a5f-de29-4751-bfb9-947f381498c4-kube-api-access-qmhtt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:08:46.714057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:46.714059 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:08:46.714270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:46.714075 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d88a5f-de29-4751-bfb9-947f381498c4-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.381711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:47.381684 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" Oct 30 06:08:47.381711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:47.381699 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq" event={"ID":"39d88a5f-de29-4751-bfb9-947f381498c4","Type":"ContainerDied","Data":"e501caa93c2f78d3f42a7aeb44096de8486ea94eb73adaa6d56f684fb539a705"} Oct 30 06:08:47.381921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:47.381748 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e501caa93c2f78d3f42a7aeb44096de8486ea94eb73adaa6d56f684fb539a705" Oct 30 06:08:51.494462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:51.494409 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:08:54.866638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.866603 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh"] Oct 30 06:08:54.867341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.867323 5001 cpu_manager.go:414] "containerMap: removing container" podUID="39d88a5f-de29-4751-bfb9-947f381498c4" containerName="revoke-keys" Oct 30 06:08:54.867402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.867343 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d88a5f-de29-4751-bfb9-947f381498c4" containerName="revoke-keys" Oct 30 06:08:54.867512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.867502 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d88a5f-de29-4751-bfb9-947f381498c4" containerName="revoke-keys" Oct 30 06:08:54.870913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.870896 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:54.891774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.891750 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh"] Oct 30 06:08:54.900220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.900157 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:08:54.900220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.900174 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:08:54.900220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.900211 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:08:54.900529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.900336 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:08:54.957789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.957754 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3b890e-tzzgh\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:54.957789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.957807 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3b890e-tzzgh\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:54.958023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:54.957969 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78lqg\" (UniqueName: \"kubernetes.io/projected/3afe7122-ca01-4c70-87c9-b03e115d8575-kube-api-access-78lqg\") pod \"maas-api-revoke-sub-ba3b890e-tzzgh\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:55.068848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:55.059643 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78lqg\" (UniqueName: \"kubernetes.io/projected/3afe7122-ca01-4c70-87c9-b03e115d8575-kube-api-access-78lqg\") pod \"maas-api-revoke-sub-ba3b890e-tzzgh\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:55.068848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:55.059783 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3b890e-tzzgh\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:55.068848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:55.059839 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3b890e-tzzgh\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:55.070676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:55.070650 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3b890e-tzzgh\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:55.070797 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:55.070726 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3b890e-tzzgh\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:55.085488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:55.085459 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78lqg\" (UniqueName: \"kubernetes.io/projected/3afe7122-ca01-4c70-87c9-b03e115d8575-kube-api-access-78lqg\") pod \"maas-api-revoke-sub-ba3b890e-tzzgh\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:55.184776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:55.184702 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:55.498064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:55.498039 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh"] Oct 30 06:08:55.500266 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:08:55.500234 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afe7122_ca01_4c70_87c9_b03e115d8575.slice/crio-1fbd337988623af585ff5df45621b3229a529c8b9defb04cd0892d749fdf3565 WatchSource:0}: Error finding container 1fbd337988623af585ff5df45621b3229a529c8b9defb04cd0892d749fdf3565: Status 404 returned error can't find the container with id 1fbd337988623af585ff5df45621b3229a529c8b9defb04cd0892d749fdf3565 Oct 30 06:08:56.446176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:56.446145 5001 generic.go:356] "Generic (PLEG): container finished" podID="3afe7122-ca01-4c70-87c9-b03e115d8575" containerID="7fee3710cbf570491fdc240b3198b9cb6ba7b618607ffe9136c01203ddfb12de" exitCode=0 Oct 30 06:08:56.446576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:56.446226 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" event={"ID":"3afe7122-ca01-4c70-87c9-b03e115d8575","Type":"ContainerDied","Data":"7fee3710cbf570491fdc240b3198b9cb6ba7b618607ffe9136c01203ddfb12de"} Oct 30 06:08:56.446576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:56.446278 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" event={"ID":"3afe7122-ca01-4c70-87c9-b03e115d8575","Type":"ContainerStarted","Data":"1fbd337988623af585ff5df45621b3229a529c8b9defb04cd0892d749fdf3565"} Oct 30 06:08:57.613156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.613128 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:57.688775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.688746 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afe7122-ca01-4c70-87c9-b03e115d8575-kube-api-access-78lqg\" (UniqueName: \"kubernetes.io/projected/3afe7122-ca01-4c70-87c9-b03e115d8575-kube-api-access-78lqg\") pod \"3afe7122-ca01-4c70-87c9-b03e115d8575\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " Oct 30 06:08:57.688955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.688860 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-xks-ca\") pod \"3afe7122-ca01-4c70-87c9-b03e115d8575\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " Oct 30 06:08:57.688955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.688908 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-openshift-ca\") pod \"3afe7122-ca01-4c70-87c9-b03e115d8575\" (UID: \"3afe7122-ca01-4c70-87c9-b03e115d8575\") " Oct 30 06:08:57.689363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.689326 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-openshift-ca" pod "3afe7122-ca01-4c70-87c9-b03e115d8575" (UID: "3afe7122-ca01-4c70-87c9-b03e115d8575"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:57.691512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.691482 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-xks-ca" pod "3afe7122-ca01-4c70-87c9-b03e115d8575" (UID: "3afe7122-ca01-4c70-87c9-b03e115d8575"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:57.691634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.691542 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afe7122-ca01-4c70-87c9-b03e115d8575-kube-api-access-78lqg" pod "3afe7122-ca01-4c70-87c9-b03e115d8575" (UID: "3afe7122-ca01-4c70-87c9-b03e115d8575"). InnerVolumeSpecName "kube-api-access-78lqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:57.789962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.789926 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:08:57.789962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.789959 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afe7122-ca01-4c70-87c9-b03e115d8575-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:08:57.789962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.789970 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78lqg\" (UniqueName: \"kubernetes.io/projected/3afe7122-ca01-4c70-87c9-b03e115d8575-kube-api-access-78lqg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:08:57.962755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.962659 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz"] Oct 30 06:08:57.963199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.963182 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3afe7122-ca01-4c70-87c9-b03e115d8575" containerName="revoke-keys" Oct 30 06:08:57.963199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.963197 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afe7122-ca01-4c70-87c9-b03e115d8575" containerName="revoke-keys" Oct 30 06:08:57.963399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.963332 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afe7122-ca01-4c70-87c9-b03e115d8575" containerName="revoke-keys" Oct 30 06:08:57.966845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.966822 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:08:57.989272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:57.989239 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz"] Oct 30 06:08:58.093977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.093946 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5d795c-t8kdz\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:08:58.094147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.093992 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf95j\" (UniqueName: \"kubernetes.io/projected/269b438f-e946-4b57-b782-d428128c2c9a-kube-api-access-vf95j\") pod \"maas-api-revoke-sub-3e5d795c-t8kdz\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:08:58.094147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.094119 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5d795c-t8kdz\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:08:58.195484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.195438 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf95j\" (UniqueName: \"kubernetes.io/projected/269b438f-e946-4b57-b782-d428128c2c9a-kube-api-access-vf95j\") pod \"maas-api-revoke-sub-3e5d795c-t8kdz\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:08:58.195678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.195561 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5d795c-t8kdz\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:08:58.195731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.195710 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5d795c-t8kdz\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:08:58.196440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.196409 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5d795c-t8kdz\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:08:58.203330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.200692 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5d795c-t8kdz\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:08:58.219984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.219916 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf95j\" (UniqueName: \"kubernetes.io/projected/269b438f-e946-4b57-b782-d428128c2c9a-kube-api-access-vf95j\") pod \"maas-api-revoke-sub-3e5d795c-t8kdz\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:08:58.281800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.281768 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:08:58.462508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.462481 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" Oct 30 06:08:58.462666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.462513 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh" event={"ID":"3afe7122-ca01-4c70-87c9-b03e115d8575","Type":"ContainerDied","Data":"1fbd337988623af585ff5df45621b3229a529c8b9defb04cd0892d749fdf3565"} Oct 30 06:08:58.462666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.462586 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fbd337988623af585ff5df45621b3229a529c8b9defb04cd0892d749fdf3565" Oct 30 06:08:58.594090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.594067 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz"] Oct 30 06:08:58.596988 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:08:58.596957 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269b438f_e946_4b57_b782_d428128c2c9a.slice/crio-3bacaaac77ff73c2f2f70661fc83849d50e2606be217fb2c502770cbd76d1ab4 WatchSource:0}: Error finding container 3bacaaac77ff73c2f2f70661fc83849d50e2606be217fb2c502770cbd76d1ab4: Status 404 returned error can't find the container with id 3bacaaac77ff73c2f2f70661fc83849d50e2606be217fb2c502770cbd76d1ab4 Oct 30 06:08:58.755749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.755716 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg"] Oct 30 06:08:58.761098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.761077 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:08:58.779598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.779564 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg"] Oct 30 06:08:58.903174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.903141 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46dbef1c-rsbqg\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:08:58.903287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.903229 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46dbef1c-rsbqg\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:08:58.903392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:58.903372 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxj9t\" (UniqueName: \"kubernetes.io/projected/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-kube-api-access-gxj9t\") pod \"maas-api-revoke-sub-46dbef1c-rsbqg\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:08:59.004347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.004293 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46dbef1c-rsbqg\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:08:59.004544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.004375 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxj9t\" (UniqueName: \"kubernetes.io/projected/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-kube-api-access-gxj9t\") pod \"maas-api-revoke-sub-46dbef1c-rsbqg\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:08:59.004544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.004421 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46dbef1c-rsbqg\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:08:59.005017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.004997 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46dbef1c-rsbqg\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:08:59.006835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.006815 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46dbef1c-rsbqg\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:08:59.028365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.028342 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxj9t\" (UniqueName: \"kubernetes.io/projected/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-kube-api-access-gxj9t\") pod \"maas-api-revoke-sub-46dbef1c-rsbqg\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:08:59.078450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.078418 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:08:59.414526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.414496 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg"] Oct 30 06:08:59.430124 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:08:59.430088 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60f6d92_1eef_463f_a9a3_2fcdd6d0cb54.slice/crio-6cee87007f7514640885161070cd678f135009ff4701beb33f9d9823c9292321 WatchSource:0}: Error finding container 6cee87007f7514640885161070cd678f135009ff4701beb33f9d9823c9292321: Status 404 returned error can't find the container with id 6cee87007f7514640885161070cd678f135009ff4701beb33f9d9823c9292321 Oct 30 06:08:59.470779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.470747 5001 generic.go:356] "Generic (PLEG): container finished" podID="269b438f-e946-4b57-b782-d428128c2c9a" containerID="bfc0d53448bd9d2e7f4e11c85bbfd3fdf338c2069f98aca3af882d5b2bba5d17" exitCode=0 Oct 30 06:08:59.470915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.470840 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" event={"ID":"269b438f-e946-4b57-b782-d428128c2c9a","Type":"ContainerDied","Data":"bfc0d53448bd9d2e7f4e11c85bbfd3fdf338c2069f98aca3af882d5b2bba5d17"} Oct 30 06:08:59.470915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.470902 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" event={"ID":"269b438f-e946-4b57-b782-d428128c2c9a","Type":"ContainerStarted","Data":"3bacaaac77ff73c2f2f70661fc83849d50e2606be217fb2c502770cbd76d1ab4"} Oct 30 06:08:59.472337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:08:59.472291 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" event={"ID":"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54","Type":"ContainerStarted","Data":"6cee87007f7514640885161070cd678f135009ff4701beb33f9d9823c9292321"} Oct 30 06:09:00.481709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.481678 5001 generic.go:356] "Generic (PLEG): container finished" podID="b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54" containerID="d2bdf61371c11db882a7f94258a23d403dfd2405eb98ed4637cb9312ecb0b11b" exitCode=0 Oct 30 06:09:00.482108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.481758 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" event={"ID":"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54","Type":"ContainerDied","Data":"d2bdf61371c11db882a7f94258a23d403dfd2405eb98ed4637cb9312ecb0b11b"} Oct 30 06:09:00.644003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.643978 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:09:00.722356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.722257 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-xks-ca\") pod \"269b438f-e946-4b57-b782-d428128c2c9a\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " Oct 30 06:09:00.722356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.722297 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269b438f-e946-4b57-b782-d428128c2c9a-kube-api-access-vf95j\" (UniqueName: \"kubernetes.io/projected/269b438f-e946-4b57-b782-d428128c2c9a-kube-api-access-vf95j\") pod \"269b438f-e946-4b57-b782-d428128c2c9a\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " Oct 30 06:09:00.722554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.722394 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-openshift-ca\") pod \"269b438f-e946-4b57-b782-d428128c2c9a\" (UID: \"269b438f-e946-4b57-b782-d428128c2c9a\") " Oct 30 06:09:00.722786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.722757 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-openshift-ca" pod "269b438f-e946-4b57-b782-d428128c2c9a" (UID: "269b438f-e946-4b57-b782-d428128c2c9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:00.724855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.724786 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-xks-ca" pod "269b438f-e946-4b57-b782-d428128c2c9a" (UID: "269b438f-e946-4b57-b782-d428128c2c9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:00.725197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.724957 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269b438f-e946-4b57-b782-d428128c2c9a-kube-api-access-vf95j" pod "269b438f-e946-4b57-b782-d428128c2c9a" (UID: "269b438f-e946-4b57-b782-d428128c2c9a"). InnerVolumeSpecName "kube-api-access-vf95j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:00.823571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.823536 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:00.823571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.823569 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269b438f-e946-4b57-b782-d428128c2c9a-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:00.823571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:00.823581 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf95j\" (UniqueName: \"kubernetes.io/projected/269b438f-e946-4b57-b782-d428128c2c9a-kube-api-access-vf95j\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:01.495337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:01.494830 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" Oct 30 06:09:01.514350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:01.504446 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz" event={"ID":"269b438f-e946-4b57-b782-d428128c2c9a","Type":"ContainerDied","Data":"3bacaaac77ff73c2f2f70661fc83849d50e2606be217fb2c502770cbd76d1ab4"} Oct 30 06:09:01.514350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:01.504548 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bacaaac77ff73c2f2f70661fc83849d50e2606be217fb2c502770cbd76d1ab4" Oct 30 06:09:01.898946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:01.897566 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:09:02.012939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.012904 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-openshift-ca\") pod \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " Oct 30 06:09:02.013135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.013032 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-kube-api-access-gxj9t\" (UniqueName: \"kubernetes.io/projected/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-kube-api-access-gxj9t\") pod \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " Oct 30 06:09:02.013135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.013066 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-xks-ca\") pod \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\" (UID: \"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54\") " Oct 30 06:09:02.013484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.013288 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-openshift-ca" pod "b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54" (UID: "b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:02.013569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.013535 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:02.015259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.015233 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-kube-api-access-gxj9t" pod "b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54" (UID: "b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54"). InnerVolumeSpecName "kube-api-access-gxj9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:02.015400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.015289 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-xks-ca" pod "b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54" (UID: "b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:02.114399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.114322 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxj9t\" (UniqueName: \"kubernetes.io/projected/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-kube-api-access-gxj9t\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:02.114399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.114360 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:02.503248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.503156 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" Oct 30 06:09:02.503248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.503193 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg" event={"ID":"b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54","Type":"ContainerDied","Data":"6cee87007f7514640885161070cd678f135009ff4701beb33f9d9823c9292321"} Oct 30 06:09:02.503248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:02.503245 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cee87007f7514640885161070cd678f135009ff4701beb33f9d9823c9292321" Oct 30 06:09:07.486688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.486651 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2"] Oct 30 06:09:07.487275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.487254 5001 cpu_manager.go:414] "containerMap: removing container" podUID="269b438f-e946-4b57-b782-d428128c2c9a" containerName="revoke-keys" Oct 30 06:09:07.487275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.487275 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269b438f-e946-4b57-b782-d428128c2c9a" containerName="revoke-keys" Oct 30 06:09:07.487491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.487340 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54" containerName="revoke-keys" Oct 30 06:09:07.487491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.487350 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54" containerName="revoke-keys" Oct 30 06:09:07.487491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.487487 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54" containerName="revoke-keys" Oct 30 06:09:07.487640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.487506 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="269b438f-e946-4b57-b782-d428128c2c9a" containerName="revoke-keys" Oct 30 06:09:07.491210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.491187 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.510134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.510106 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8d44dd-openshift-default-dockercfg-nzqwp\"" Oct 30 06:09:07.531908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.531859 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2"] Oct 30 06:09:07.573494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.573450 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrln\" (UniqueName: \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-kube-api-access-xlrln\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.573688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.573583 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.573688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.573620 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.573808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.573691 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.573808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.573727 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.573918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.573850 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.573972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.573913 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ce104-9403-4712-8512-9845efc5ac51-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.574022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.573976 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ce104-9403-4712-8512-9845efc5ac51-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.574022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.574002 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.675556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.675520 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ce104-9403-4712-8512-9845efc5ac51-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.675752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.675568 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.675752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.675632 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrln\" (UniqueName: \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-kube-api-access-xlrln\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.675752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.675689 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.675752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.675720 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.675752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.675743 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.676027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.675856 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.676027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.675957 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.676027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.676010 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ce104-9403-4712-8512-9845efc5ac51-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.676180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.676059 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.676180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.676098 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.676292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.676187 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.676391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.676357 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ce104-9403-4712-8512-9845efc5ac51-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.676460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.676398 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.678267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.678247 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.678589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.678573 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ce104-9403-4712-8512-9845efc5ac51-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.700300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.700272 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.700438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.700361 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrln\" (UniqueName: \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-kube-api-access-xlrln\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:07.820481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:07.820386 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:08.153299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:08.152967 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2"] Oct 30 06:09:08.156670 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:09:08.155816 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5ce104_9403_4712_8512_9845efc5ac51.slice/crio-ce22139329e95663a5dc5b5825bb5e71fec16016224d531883d279fdd65482e6 WatchSource:0}: Error finding container ce22139329e95663a5dc5b5825bb5e71fec16016224d531883d279fdd65482e6: Status 404 returned error can't find the container with id ce22139329e95663a5dc5b5825bb5e71fec16016224d531883d279fdd65482e6 Oct 30 06:09:08.161623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:08.161585 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:08.161722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:08.161674 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:08.161722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:08.161714 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:08.287851 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:09:08.287786 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:09:08.556198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:08.556085 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" event={"ID":"ad5ce104-9403-4712-8512-9845efc5ac51","Type":"ContainerStarted","Data":"7381bc7f6f15ac882500104dd3fb6dee705cc44f91cd5b3ce31950ef53a8b41b"} Oct 30 06:09:08.556198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:08.556157 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" event={"ID":"ad5ce104-9403-4712-8512-9845efc5ac51","Type":"ContainerStarted","Data":"ce22139329e95663a5dc5b5825bb5e71fec16016224d531883d279fdd65482e6"} Oct 30 06:09:08.623724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:08.623672 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" podStartSLOduration=1.623659166 podStartE2EDuration="1.623659166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:08.622662113 +0000 UTC m=+962.618619124" watchObservedRunningTime="2026-10-30 06:09:08.623659166 +0000 UTC m=+962.619616175" Oct 30 06:09:08.820990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:08.820872 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:09.822695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:09.822661 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:09:09.823093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:09.822719 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" podUID="ad5ce104-9403-4712-8512-9845efc5ac51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 06:09:10.821027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:10.820987 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:09:10.821218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:10.821062 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" podUID="ad5ce104-9403-4712-8512-9845efc5ac51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 06:09:11.821220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:11.821189 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:09:11.821686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:11.821254 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" podUID="ad5ce104-9403-4712-8512-9845efc5ac51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 06:09:12.821054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:12.821017 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:09:12.821238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:12.821068 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" podUID="ad5ce104-9403-4712-8512-9845efc5ac51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 06:09:13.297210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.297173 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd"] Oct 30 06:09:13.301280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.301256 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:13.320651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.320630 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd"] Oct 30 06:09:13.330033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.329891 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:09:13.330033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.329943 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:09:13.330033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.330001 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:09:13.330370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.330354 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:09:13.339641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.339607 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-505032de-297pd\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:13.339787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.339771 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-505032de-297pd\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:13.339870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.339817 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjtb\" (UniqueName: \"kubernetes.io/projected/1b83e190-4707-4fb6-b93e-f8304c6edf9c-kube-api-access-5qjtb\") pod \"maas-api-revoke-sub-505032de-297pd\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:13.440998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.440967 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjtb\" (UniqueName: \"kubernetes.io/projected/1b83e190-4707-4fb6-b93e-f8304c6edf9c-kube-api-access-5qjtb\") pod \"maas-api-revoke-sub-505032de-297pd\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:13.441168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.441008 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-505032de-297pd\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:13.441168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.441108 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-505032de-297pd\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:13.441703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.441685 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-505032de-297pd\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:13.443508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.443482 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-505032de-297pd\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:13.478485 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.478459 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjtb\" (UniqueName: \"kubernetes.io/projected/1b83e190-4707-4fb6-b93e-f8304c6edf9c-kube-api-access-5qjtb\") pod \"maas-api-revoke-sub-505032de-297pd\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:13.615024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.614895 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:13.820843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.820809 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:09:13.821012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.820866 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" podUID="ad5ce104-9403-4712-8512-9845efc5ac51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 06:09:13.965035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:13.964917 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd"] Oct 30 06:09:14.627640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:14.627608 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" event={"ID":"1b83e190-4707-4fb6-b93e-f8304c6edf9c","Type":"ContainerStarted","Data":"a28c9cf08a9c603d73d33f5cf28d6311e4fb06624c2c1c8041d585ca7b0fbc0f"} Oct 30 06:09:14.628123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:14.627659 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" event={"ID":"1b83e190-4707-4fb6-b93e-f8304c6edf9c","Type":"ContainerStarted","Data":"58fe5e6df53c79ea1dea6d85251961a5bda4e05e37239393279584ccd9688855"} Oct 30 06:09:14.671591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:14.671533 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" podStartSLOduration=1.671512155 podStartE2EDuration="1.671512155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:14.670366931 +0000 UTC m=+968.666323943" watchObservedRunningTime="2026-10-30 06:09:14.671512155 +0000 UTC m=+968.667469166" Oct 30 06:09:15.040558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:15.040524 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:15.041643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:15.041610 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:15.042936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:15.042910 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:09:15.635870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:15.635834 5001 generic.go:356] "Generic (PLEG): container finished" podID="1b83e190-4707-4fb6-b93e-f8304c6edf9c" containerID="a28c9cf08a9c603d73d33f5cf28d6311e4fb06624c2c1c8041d585ca7b0fbc0f" exitCode=0 Oct 30 06:09:15.636386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:15.635907 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" event={"ID":"1b83e190-4707-4fb6-b93e-f8304c6edf9c","Type":"ContainerDied","Data":"a28c9cf08a9c603d73d33f5cf28d6311e4fb06624c2c1c8041d585ca7b0fbc0f"} Oct 30 06:09:16.806671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:16.806646 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:16.887954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:16.887915 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-xks-ca\") pod \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " Oct 30 06:09:16.888127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:16.888036 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-openshift-ca\") pod \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " Oct 30 06:09:16.888127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:16.888102 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b83e190-4707-4fb6-b93e-f8304c6edf9c-kube-api-access-5qjtb\" (UniqueName: \"kubernetes.io/projected/1b83e190-4707-4fb6-b93e-f8304c6edf9c-kube-api-access-5qjtb\") pod \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\" (UID: \"1b83e190-4707-4fb6-b93e-f8304c6edf9c\") " Oct 30 06:09:16.888518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:16.888447 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-openshift-ca" pod "1b83e190-4707-4fb6-b93e-f8304c6edf9c" (UID: "1b83e190-4707-4fb6-b93e-f8304c6edf9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:16.890886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:16.890860 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b83e190-4707-4fb6-b93e-f8304c6edf9c-kube-api-access-5qjtb" pod "1b83e190-4707-4fb6-b93e-f8304c6edf9c" (UID: "1b83e190-4707-4fb6-b93e-f8304c6edf9c"). InnerVolumeSpecName "kube-api-access-5qjtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:16.901693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:16.901635 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-xks-ca" pod "1b83e190-4707-4fb6-b93e-f8304c6edf9c" (UID: "1b83e190-4707-4fb6-b93e-f8304c6edf9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:16.990608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:16.989426 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:16.990608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:16.989457 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qjtb\" (UniqueName: \"kubernetes.io/projected/1b83e190-4707-4fb6-b93e-f8304c6edf9c-kube-api-access-5qjtb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:16.990608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:16.989467 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b83e190-4707-4fb6-b93e-f8304c6edf9c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:17.650764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:17.650735 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" Oct 30 06:09:17.650920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:17.650730 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd" event={"ID":"1b83e190-4707-4fb6-b93e-f8304c6edf9c","Type":"ContainerDied","Data":"58fe5e6df53c79ea1dea6d85251961a5bda4e05e37239393279584ccd9688855"} Oct 30 06:09:17.650920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:17.650865 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58fe5e6df53c79ea1dea6d85251961a5bda4e05e37239393279584ccd9688855" Oct 30 06:09:23.754317 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:23.754262 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:09:26.456204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.456164 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd"] Oct 30 06:09:26.456841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.456729 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1b83e190-4707-4fb6-b93e-f8304c6edf9c" containerName="revoke-keys" Oct 30 06:09:26.456841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.456743 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b83e190-4707-4fb6-b93e-f8304c6edf9c" containerName="revoke-keys" Oct 30 06:09:26.456841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.456835 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b83e190-4707-4fb6-b93e-f8304c6edf9c" containerName="revoke-keys" Oct 30 06:09:26.460404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.460384 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:26.481542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.481495 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd"] Oct 30 06:09:26.490563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.490536 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:09:26.490818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.490795 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:09:26.490976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.490822 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:09:26.492486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.492465 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:09:26.641087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.641053 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca0315c1-hl7vd\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:26.641087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.641089 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4g7v\" (UniqueName: \"kubernetes.io/projected/c73cb86f-83d8-492b-8a2a-41027668e826-kube-api-access-d4g7v\") pod \"maas-api-revoke-sub-ca0315c1-hl7vd\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:26.641409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.641195 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca0315c1-hl7vd\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:26.742427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.742336 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca0315c1-hl7vd\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:26.742611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.742454 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca0315c1-hl7vd\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:26.742611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.742475 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4g7v\" (UniqueName: \"kubernetes.io/projected/c73cb86f-83d8-492b-8a2a-41027668e826-kube-api-access-d4g7v\") pod \"maas-api-revoke-sub-ca0315c1-hl7vd\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:26.743133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.743107 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca0315c1-hl7vd\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:26.744787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.744768 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca0315c1-hl7vd\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:26.766351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.766329 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4g7v\" (UniqueName: \"kubernetes.io/projected/c73cb86f-83d8-492b-8a2a-41027668e826-kube-api-access-d4g7v\") pod \"maas-api-revoke-sub-ca0315c1-hl7vd\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:26.778173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:26.778150 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:27.082421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:27.082398 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd"] Oct 30 06:09:27.083261 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:09:27.083236 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73cb86f_83d8_492b_8a2a_41027668e826.slice/crio-ac742cd7d9643bc2a6bd1ac61feabdec01db9d61592e3cc50ac94df6ec646a29 WatchSource:0}: Error finding container ac742cd7d9643bc2a6bd1ac61feabdec01db9d61592e3cc50ac94df6ec646a29: Status 404 returned error can't find the container with id ac742cd7d9643bc2a6bd1ac61feabdec01db9d61592e3cc50ac94df6ec646a29 Oct 30 06:09:27.733527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:27.733441 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" event={"ID":"c73cb86f-83d8-492b-8a2a-41027668e826","Type":"ContainerStarted","Data":"c63bb32308c7cb9f3ffd6b7a63a7f99a854deecf880a4c6c7cbd3d4fd3e24015"} Oct 30 06:09:27.733527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:27.733500 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" event={"ID":"c73cb86f-83d8-492b-8a2a-41027668e826","Type":"ContainerStarted","Data":"ac742cd7d9643bc2a6bd1ac61feabdec01db9d61592e3cc50ac94df6ec646a29"} Oct 30 06:09:27.778721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:27.778670 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" podStartSLOduration=1.778655075 podStartE2EDuration="1.778655075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:27.776991723 +0000 UTC m=+981.772948733" watchObservedRunningTime="2026-10-30 06:09:27.778655075 +0000 UTC m=+981.774612116" Oct 30 06:09:28.741436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:28.741405 5001 generic.go:356] "Generic (PLEG): container finished" podID="c73cb86f-83d8-492b-8a2a-41027668e826" containerID="c63bb32308c7cb9f3ffd6b7a63a7f99a854deecf880a4c6c7cbd3d4fd3e24015" exitCode=0 Oct 30 06:09:28.741913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:28.741494 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" event={"ID":"c73cb86f-83d8-492b-8a2a-41027668e826","Type":"ContainerDied","Data":"c63bb32308c7cb9f3ffd6b7a63a7f99a854deecf880a4c6c7cbd3d4fd3e24015"} Oct 30 06:09:29.908715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:29.908678 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:29.985297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:29.985267 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-openshift-ca\") pod \"c73cb86f-83d8-492b-8a2a-41027668e826\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " Oct 30 06:09:29.985504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:29.985323 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73cb86f-83d8-492b-8a2a-41027668e826-kube-api-access-d4g7v\" (UniqueName: \"kubernetes.io/projected/c73cb86f-83d8-492b-8a2a-41027668e826-kube-api-access-d4g7v\") pod \"c73cb86f-83d8-492b-8a2a-41027668e826\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " Oct 30 06:09:29.985504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:29.985390 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-xks-ca\") pod \"c73cb86f-83d8-492b-8a2a-41027668e826\" (UID: \"c73cb86f-83d8-492b-8a2a-41027668e826\") " Oct 30 06:09:29.985723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:29.985697 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-openshift-ca" pod "c73cb86f-83d8-492b-8a2a-41027668e826" (UID: "c73cb86f-83d8-492b-8a2a-41027668e826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:29.985859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:29.985840 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:29.988569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:29.988538 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-xks-ca" pod "c73cb86f-83d8-492b-8a2a-41027668e826" (UID: "c73cb86f-83d8-492b-8a2a-41027668e826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:29.988569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:29.988543 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73cb86f-83d8-492b-8a2a-41027668e826-kube-api-access-d4g7v" pod "c73cb86f-83d8-492b-8a2a-41027668e826" (UID: "c73cb86f-83d8-492b-8a2a-41027668e826"). InnerVolumeSpecName "kube-api-access-d4g7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:30.086741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:30.086663 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4g7v\" (UniqueName: \"kubernetes.io/projected/c73cb86f-83d8-492b-8a2a-41027668e826-kube-api-access-d4g7v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:30.086741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:30.086693 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73cb86f-83d8-492b-8a2a-41027668e826-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:09:30.756945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:30.756909 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" event={"ID":"c73cb86f-83d8-492b-8a2a-41027668e826","Type":"ContainerDied","Data":"ac742cd7d9643bc2a6bd1ac61feabdec01db9d61592e3cc50ac94df6ec646a29"} Oct 30 06:09:30.757135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:30.756972 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac742cd7d9643bc2a6bd1ac61feabdec01db9d61592e3cc50ac94df6ec646a29" Oct 30 06:09:30.757135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:30.756927 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd" Oct 30 06:09:38.280637 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:09:38.280593 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:09:40.540973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:40.540940 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:09:55.328743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.328712 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw"] Oct 30 06:09:55.329376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.329326 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c73cb86f-83d8-492b-8a2a-41027668e826" containerName="revoke-keys" Oct 30 06:09:55.329376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.329347 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73cb86f-83d8-492b-8a2a-41027668e826" containerName="revoke-keys" Oct 30 06:09:55.329612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.329539 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73cb86f-83d8-492b-8a2a-41027668e826" containerName="revoke-keys" Oct 30 06:09:55.333917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.333885 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.337315 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.337264 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw"] Oct 30 06:09:55.349767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.349748 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"e2e-conflict-b-ce7e342c-kserve-self-signed-certs\"" Oct 30 06:09:55.379851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.379825 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-dshm\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.380114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.380089 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-home\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.380242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.380128 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-tmp-dir\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.380242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.380151 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672125e5-94a5-4e3c-8cde-644966e57e0f-tls-certs\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.380242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.380226 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxc4k\" (UniqueName: \"kubernetes.io/projected/672125e5-94a5-4e3c-8cde-644966e57e0f-kube-api-access-pxc4k\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.380503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.380478 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-model-cache\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.497332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.487933 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-model-cache\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.497332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.488067 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-dshm\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.497332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.488155 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-home\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.497332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.488180 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-tmp-dir\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.497332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.488202 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672125e5-94a5-4e3c-8cde-644966e57e0f-tls-certs\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.497332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.488235 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxc4k\" (UniqueName: \"kubernetes.io/projected/672125e5-94a5-4e3c-8cde-644966e57e0f-kube-api-access-pxc4k\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.497332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.494157 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-home\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.497332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.494531 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-model-cache\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.497332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.495118 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-tmp-dir\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.517941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.510138 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-dshm\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.517941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.513538 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672125e5-94a5-4e3c-8cde-644966e57e0f-tls-certs\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.538111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.538078 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxc4k\" (UniqueName: \"kubernetes.io/projected/672125e5-94a5-4e3c-8cde-644966e57e0f-kube-api-access-pxc4k\") pod \"e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.648408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.648377 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:09:55.958174 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:09:55.958140 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod672125e5_94a5_4e3c_8cde_644966e57e0f.slice/crio-7afb1871122e44e6200afefc7560f51e2286be6ac3282ab25e9f1ed3869819f6 WatchSource:0}: Error finding container 7afb1871122e44e6200afefc7560f51e2286be6ac3282ab25e9f1ed3869819f6: Status 404 returned error can't find the container with id 7afb1871122e44e6200afefc7560f51e2286be6ac3282ab25e9f1ed3869819f6 Oct 30 06:09:55.958174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:55.958161 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw"] Oct 30 06:09:56.979133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:56.979096 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" event={"ID":"672125e5-94a5-4e3c-8cde-644966e57e0f","Type":"ContainerStarted","Data":"3849c40becf4bbdc132586b0d4e8c1cb838beb6e2c3e9b687f3f8d2d2baa57f6"} Oct 30 06:09:56.979763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:56.979150 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" event={"ID":"672125e5-94a5-4e3c-8cde-644966e57e0f","Type":"ContainerStarted","Data":"7afb1871122e44e6200afefc7560f51e2286be6ac3282ab25e9f1ed3869819f6"} Oct 30 06:09:57.024640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:57.024586 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" podStartSLOduration=2.024570836 podStartE2EDuration="2.024570836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:57.023249504 +0000 UTC m=+1011.019206538" watchObservedRunningTime="2026-10-30 06:09:57.024570836 +0000 UTC m=+1011.020527878" Oct 30 06:09:57.649289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:09:57.649251 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:10:00.128712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:00.128680 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:10:05.649484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:05.649445 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:10:05.669083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:05.669052 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:10:06.056590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:06.056507 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:10:06.982607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:06.982569 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g"] Oct 30 06:10:06.987627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:06.987600 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.004810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.004782 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g"] Oct 30 06:10:07.014378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.014350 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"distinct-2-w3-8d6349-kserve-self-signed-certs\"" Oct 30 06:10:07.014378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.014368 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"default-dockercfg-xkpcw\"" Oct 30 06:10:07.017134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.017113 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"openshift-service-ca.crt\"" Oct 30 06:10:07.017251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.017160 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"kube-root-ca.crt\"" Oct 30 06:10:07.126883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.126854 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tmp-dir\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.127106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.126907 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7k7v\" (UniqueName: \"kubernetes.io/projected/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-kube-api-access-d7k7v\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.127106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.126934 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tls-certs\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.127106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.127037 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-dshm\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.127278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.127163 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-home\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.127369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.127329 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-model-cache\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.227989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.227958 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tmp-dir\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.228195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.228010 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7k7v\" (UniqueName: \"kubernetes.io/projected/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-kube-api-access-d7k7v\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.228195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.228054 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tls-certs\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.228195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.228071 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-dshm\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.228195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.228108 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-home\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.228195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.228149 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-model-cache\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.228526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.228461 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tmp-dir\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.228526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.228515 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-model-cache\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.228742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.228721 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-home\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.230765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.230745 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-dshm\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.230895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.230874 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tls-certs\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.253409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.253340 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7k7v\" (UniqueName: \"kubernetes.io/projected/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-kube-api-access-d7k7v\") pod \"distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.304480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.304446 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:07.609440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:07.609406 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g"] Oct 30 06:10:07.611007 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:10:07.610979 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73e78f9f_36ac_4ec3_a7ce_da47e56efc17.slice/crio-bc2f4d2a1307d6855e1bb7903cbc0810be0b384e37fda6db4a4f0434f11c520e WatchSource:0}: Error finding container bc2f4d2a1307d6855e1bb7903cbc0810be0b384e37fda6db4a4f0434f11c520e: Status 404 returned error can't find the container with id bc2f4d2a1307d6855e1bb7903cbc0810be0b384e37fda6db4a4f0434f11c520e Oct 30 06:10:08.060805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:08.060707 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" event={"ID":"73e78f9f-36ac-4ec3-a7ce-da47e56efc17","Type":"ContainerStarted","Data":"6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185"} Oct 30 06:10:08.060805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:08.060760 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" event={"ID":"73e78f9f-36ac-4ec3-a7ce-da47e56efc17","Type":"ContainerStarted","Data":"bc2f4d2a1307d6855e1bb7903cbc0810be0b384e37fda6db4a4f0434f11c520e"} Oct 30 06:10:08.104658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:08.104609 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podStartSLOduration=2.104594168 podStartE2EDuration="2.104594168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:08.103844036 +0000 UTC m=+1022.099801049" watchObservedRunningTime="2026-10-30 06:10:08.104594168 +0000 UTC m=+1022.100551211" Oct 30 06:10:08.293705 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:10:08.293667 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:10:08.304709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:08.304669 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:17.206803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.206693 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx"] Oct 30 06:10:17.210628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.210609 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.242804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.242770 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"openshift-service-ca.crt\"" Oct 30 06:10:17.242977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.242789 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"kube-root-ca.crt\"" Oct 30 06:10:17.242977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.242906 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"distinct-2-w2-aaf467-kserve-self-signed-certs\"" Oct 30 06:10:17.243168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.243145 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"default-dockercfg-sjppv\"" Oct 30 06:10:17.248199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.248173 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx"] Oct 30 06:10:17.304976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.304940 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:17.318074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.318042 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:17.356156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.356115 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-tmp-dir\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.356376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.356179 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-model-cache\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.356376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.356245 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdr7\" (UniqueName: \"kubernetes.io/projected/154bbb27-a34a-4131-a586-e724c31118b6-kube-api-access-6gdr7\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.356487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.356379 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154bbb27-a34a-4131-a586-e724c31118b6-tls-certs\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.356487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.356434 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-home\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.356599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.356498 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-dshm\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.462067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.461974 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-tmp-dir\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.462675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.462653 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-model-cache\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.463133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.463119 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdr7\" (UniqueName: \"kubernetes.io/projected/154bbb27-a34a-4131-a586-e724c31118b6-kube-api-access-6gdr7\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.463609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.463589 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154bbb27-a34a-4131-a586-e724c31118b6-tls-certs\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.463798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.463782 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-home\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.463911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.463887 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-dshm\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.464493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.463070 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-model-cache\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.465552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.462554 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-tmp-dir\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.465673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.465222 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-home\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.475453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.475435 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154bbb27-a34a-4131-a586-e724c31118b6-tls-certs\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.480633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.480608 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-dshm\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.489374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.489353 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdr7\" (UniqueName: \"kubernetes.io/projected/154bbb27-a34a-4131-a586-e724c31118b6-kube-api-access-6gdr7\") pod \"distinct-2-w2-aaf467-kserve-cfccb47-qgfvx\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.525329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.525272 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:17.854552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:17.854202 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx"] Oct 30 06:10:18.133514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:18.133473 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" event={"ID":"154bbb27-a34a-4131-a586-e724c31118b6","Type":"ContainerStarted","Data":"8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34"} Oct 30 06:10:18.133720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:18.133531 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" event={"ID":"154bbb27-a34a-4131-a586-e724c31118b6","Type":"ContainerStarted","Data":"812b21592bff45da023b09bdb95fa632888aa5d38ba4f43a225aecf9e50bbcec"} Oct 30 06:10:18.145431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:18.145402 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:10:18.179103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:18.179055 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podStartSLOduration=1.179041332 podStartE2EDuration="1.179041332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:18.176722258 +0000 UTC m=+1032.172679267" watchObservedRunningTime="2026-10-30 06:10:18.179041332 +0000 UTC m=+1032.174998342" Oct 30 06:10:18.525440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:18.525361 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:25.124124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:25.124094 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw"] Oct 30 06:10:25.724502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:25.724471 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s"] Oct 30 06:10:25.724711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:25.724675 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" containerName="istio-proxy" containerID="cri-o://f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b" gracePeriod=30 Oct 30 06:10:25.738789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:25.738755 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g"] Oct 30 06:10:25.739016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:25.738985 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" containerID="cri-o://22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16" gracePeriod=30 Oct 30 06:10:25.943787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:25.943755 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:10:27.526384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:27.526343 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:27.544771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:27.544745 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:28.227261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:28.227228 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:10:31.006131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.006110 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:10:31.022803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.022783 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:10:31.117133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117095 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-certs\") pod \"3a4973ef-1d32-41f2-bc92-286112948b10\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " Oct 30 06:10:31.117133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117136 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-kube-api-access-2b9vf\" (UniqueName: \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-kube-api-access-2b9vf\") pod \"3a4973ef-1d32-41f2-bc92-286112948b10\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " Oct 30 06:10:31.117399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117171 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-token\" (UniqueName: \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-token\") pod \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " Oct 30 06:10:31.117399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117199 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-envoy\") pod \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " Oct 30 06:10:31.117399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117237 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-podinfo\") pod \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " Oct 30 06:10:31.117399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117286 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-envoy\") pod \"3a4973ef-1d32-41f2-bc92-286112948b10\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " Oct 30 06:10:31.117399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117382 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istiod-ca-cert\") pod \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " Oct 30 06:10:31.117668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117438 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-certs\") pod \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " Oct 30 06:10:31.117668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117485 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-credential-socket\") pod \"3a4973ef-1d32-41f2-bc92-286112948b10\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " Oct 30 06:10:31.117668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117511 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-socket\") pod \"3a4973ef-1d32-41f2-bc92-286112948b10\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " Oct 30 06:10:31.117668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117569 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a4973ef-1d32-41f2-bc92-286112948b10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4973ef-1d32-41f2-bc92-286112948b10-istio-podinfo\") pod \"3a4973ef-1d32-41f2-bc92-286112948b10\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " Oct 30 06:10:31.117668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117603 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-socket\") pod \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " Oct 30 06:10:31.117668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117652 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a4973ef-1d32-41f2-bc92-286112948b10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4973ef-1d32-41f2-bc92-286112948b10-istiod-ca-cert\") pod \"3a4973ef-1d32-41f2-bc92-286112948b10\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " Oct 30 06:10:31.117961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117680 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-istio-token\" (UniqueName: \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-istio-token\") pod \"3a4973ef-1d32-41f2-bc92-286112948b10\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " Oct 30 06:10:31.117961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117716 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-kube-api-access-ck7dk\" (UniqueName: \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-kube-api-access-ck7dk\") pod \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " Oct 30 06:10:31.117961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117779 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-credential-socket\") pod \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " Oct 30 06:10:31.117961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117518 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-certs" pod "3a4973ef-1d32-41f2-bc92-286112948b10" (UID: "3a4973ef-1d32-41f2-bc92-286112948b10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:31.117961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117842 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-data\") pod \"3a4973ef-1d32-41f2-bc92-286112948b10\" (UID: \"3a4973ef-1d32-41f2-bc92-286112948b10\") " Oct 30 06:10:31.117961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.117921 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-data\") pod \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\" (UID: \"1c68d683-85ff-4411-bf67-d0f99a0c8d41\") " Oct 30 06:10:31.118648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.118551 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.118789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.118648 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-socket" pod "1c68d683-85ff-4411-bf67-d0f99a0c8d41" (UID: "1c68d683-85ff-4411-bf67-d0f99a0c8d41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:31.119240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.119043 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-credential-socket" pod "1c68d683-85ff-4411-bf67-d0f99a0c8d41" (UID: "1c68d683-85ff-4411-bf67-d0f99a0c8d41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:31.119240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.119060 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-data" pod "1c68d683-85ff-4411-bf67-d0f99a0c8d41" (UID: "1c68d683-85ff-4411-bf67-d0f99a0c8d41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:31.119519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.119260 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-credential-socket" pod "3a4973ef-1d32-41f2-bc92-286112948b10" (UID: "3a4973ef-1d32-41f2-bc92-286112948b10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:31.119519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.119465 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-certs" pod "1c68d683-85ff-4411-bf67-d0f99a0c8d41" (UID: "1c68d683-85ff-4411-bf67-d0f99a0c8d41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:31.119686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.119662 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istiod-ca-cert" pod "1c68d683-85ff-4411-bf67-d0f99a0c8d41" (UID: "1c68d683-85ff-4411-bf67-d0f99a0c8d41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:31.119914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.119862 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-data" pod "3a4973ef-1d32-41f2-bc92-286112948b10" (UID: "3a4973ef-1d32-41f2-bc92-286112948b10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:31.120432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.120125 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4973ef-1d32-41f2-bc92-286112948b10-istiod-ca-cert" pod "3a4973ef-1d32-41f2-bc92-286112948b10" (UID: "3a4973ef-1d32-41f2-bc92-286112948b10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:31.120432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.120334 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-kube-api-access-2b9vf" pod "3a4973ef-1d32-41f2-bc92-286112948b10" (UID: "3a4973ef-1d32-41f2-bc92-286112948b10"). InnerVolumeSpecName "kube-api-access-2b9vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:31.120432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.120399 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-socket" pod "3a4973ef-1d32-41f2-bc92-286112948b10" (UID: "3a4973ef-1d32-41f2-bc92-286112948b10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:31.122082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.122052 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-token" pod "1c68d683-85ff-4411-bf67-d0f99a0c8d41" (UID: "1c68d683-85ff-4411-bf67-d0f99a0c8d41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:31.122658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.122622 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a4973ef-1d32-41f2-bc92-286112948b10-istio-podinfo" pod "3a4973ef-1d32-41f2-bc92-286112948b10" (UID: "3a4973ef-1d32-41f2-bc92-286112948b10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:31.122658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.122650 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-istio-token" pod "3a4973ef-1d32-41f2-bc92-286112948b10" (UID: "3a4973ef-1d32-41f2-bc92-286112948b10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:31.122993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.122969 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-envoy" pod "1c68d683-85ff-4411-bf67-d0f99a0c8d41" (UID: "1c68d683-85ff-4411-bf67-d0f99a0c8d41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:31.122993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.122975 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-envoy" pod "3a4973ef-1d32-41f2-bc92-286112948b10" (UID: "3a4973ef-1d32-41f2-bc92-286112948b10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:31.123119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.123074 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-kube-api-access-ck7dk" pod "1c68d683-85ff-4411-bf67-d0f99a0c8d41" (UID: "1c68d683-85ff-4411-bf67-d0f99a0c8d41"). InnerVolumeSpecName "kube-api-access-ck7dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:31.123119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.123101 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-podinfo" pod "1c68d683-85ff-4411-bf67-d0f99a0c8d41" (UID: "1c68d683-85ff-4411-bf67-d0f99a0c8d41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:31.219796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219762 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4973ef-1d32-41f2-bc92-286112948b10-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.219796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219792 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.219796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219802 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck7dk\" (UniqueName: \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-kube-api-access-ck7dk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219814 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219823 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219831 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219838 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b9vf\" (UniqueName: \"kubernetes.io/projected/3a4973ef-1d32-41f2-bc92-286112948b10-kube-api-access-2b9vf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219845 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219853 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219860 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219868 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219875 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c68d683-85ff-4411-bf67-d0f99a0c8d41-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219883 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219890 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219897 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4973ef-1d32-41f2-bc92-286112948b10-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219916 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4973ef-1d32-41f2-bc92-286112948b10-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.220055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.219925 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c68d683-85ff-4411-bf67-d0f99a0c8d41-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:31.236023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.235993 5001 generic.go:356] "Generic (PLEG): container finished" podID="3a4973ef-1d32-41f2-bc92-286112948b10" containerID="f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b" exitCode=0 Oct 30 06:10:31.236228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.236041 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" Oct 30 06:10:31.236228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.236081 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" event={"ID":"3a4973ef-1d32-41f2-bc92-286112948b10","Type":"ContainerDied","Data":"f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b"} Oct 30 06:10:31.236228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.236146 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s" event={"ID":"3a4973ef-1d32-41f2-bc92-286112948b10","Type":"ContainerDied","Data":"1b403530a0e1dcbdcd69393bd1c3be378249285f28eecdda07ae7f095b56db56"} Oct 30 06:10:31.236228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.236179 5001 scope.go:122] "RemoveContainer" containerID="f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b" Oct 30 06:10:31.238127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.238104 5001 generic.go:356] "Generic (PLEG): container finished" podID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerID="22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16" exitCode=0 Oct 30 06:10:31.238220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.238192 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" event={"ID":"1c68d683-85ff-4411-bf67-d0f99a0c8d41","Type":"ContainerDied","Data":"22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16"} Oct 30 06:10:31.238220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.238200 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" Oct 30 06:10:31.238334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.238237 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g" event={"ID":"1c68d683-85ff-4411-bf67-d0f99a0c8d41","Type":"ContainerDied","Data":"59304b27ceb74e9ba1c779f916cfd6c25aec9a49556c3d6f4a88b86055327d7f"} Oct 30 06:10:31.254912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.254885 5001 scope.go:122] "RemoveContainer" containerID="f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b" Oct 30 06:10:31.255260 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:10:31.255240 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b\": container with ID starting with f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b not found: ID does not exist" containerID="f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b" Oct 30 06:10:31.255397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.255270 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b"} err="failed to get container status \"f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b\": rpc error: code = NotFound desc = could not find container \"f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b\": container with ID starting with f799e233409b7d5f442a224e7530c3915208166471bd210dd8d3d26294d7959b not found: ID does not exist" Oct 30 06:10:31.255397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.255295 5001 scope.go:122] "RemoveContainer" containerID="22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16" Oct 30 06:10:31.264948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.264925 5001 scope.go:122] "RemoveContainer" containerID="b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b" Oct 30 06:10:31.275984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.275965 5001 scope.go:122] "RemoveContainer" containerID="22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16" Oct 30 06:10:31.276329 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:10:31.276292 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16\": container with ID starting with 22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16 not found: ID does not exist" containerID="22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16" Oct 30 06:10:31.276398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.276341 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16"} err="failed to get container status \"22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16\": rpc error: code = NotFound desc = could not find container \"22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16\": container with ID starting with 22887ded6fd4706d251c1f38fe0e58a351bf929a024cc6060abcb89c8d2ebe16 not found: ID does not exist" Oct 30 06:10:31.276398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.276362 5001 scope.go:122] "RemoveContainer" containerID="b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b" Oct 30 06:10:31.276604 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:10:31.276587 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b\": container with ID starting with b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b not found: ID does not exist" containerID="b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b" Oct 30 06:10:31.276646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.276610 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b"} err="failed to get container status \"b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b\": rpc error: code = NotFound desc = could not find container \"b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b\": container with ID starting with b115ccca34035476d377e7ef3f2473facea29b423e311de1ba68c9ba9cc3018b not found: ID does not exist" Oct 30 06:10:31.292138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.292111 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s"] Oct 30 06:10:31.295000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.294975 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-28h4s"] Oct 30 06:10:31.333608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.333575 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g"] Oct 30 06:10:31.336550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.336524 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-l5d6g"] Oct 30 06:10:31.453104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.453063 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q"] Oct 30 06:10:31.453888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.453866 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" Oct 30 06:10:31.453888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.453888 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" Oct 30 06:10:31.454056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.453921 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" Oct 30 06:10:31.454056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.453929 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" Oct 30 06:10:31.454056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.453947 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" containerName="istio-proxy" Oct 30 06:10:31.454056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.453955 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" containerName="istio-proxy" Oct 30 06:10:31.454182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.454083 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" Oct 30 06:10:31.454182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.454094 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" containerName="istio-proxy" Oct 30 06:10:31.454182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.454107 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" containerName="istio-proxy" Oct 30 06:10:31.462594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.462567 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.479177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.479033 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-eaee88\"/\"openshift-service-ca.crt\"" Oct 30 06:10:31.479177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.479093 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-eaee88\"/\"kube-root-ca.crt\"" Oct 30 06:10:31.479177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.479150 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-eaee88\"/\"default-dockercfg-5fqvm\"" Oct 30 06:10:31.479177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.479151 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-eaee88\"/\"distinct-2-w4-eaee88-kserve-self-signed-certs\"" Oct 30 06:10:31.480820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.480795 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q"] Oct 30 06:10:31.522936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.522854 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-dshm\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.523080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.522996 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8112df-d5d4-41e4-adb2-560640e069b9-tls-certs\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.523080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.523067 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzjk\" (UniqueName: \"kubernetes.io/projected/8e8112df-d5d4-41e4-adb2-560640e069b9-kube-api-access-szzjk\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.523173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.523152 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-model-cache\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.523248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.523224 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-home\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.523336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.523292 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-tmp-dir\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.623989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.623956 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzjk\" (UniqueName: \"kubernetes.io/projected/8e8112df-d5d4-41e4-adb2-560640e069b9-kube-api-access-szzjk\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.624196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.624005 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-model-cache\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.624196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.624034 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-home\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.624196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.624060 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-tmp-dir\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.624196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.624101 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-dshm\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.624196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.624184 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8112df-d5d4-41e4-adb2-560640e069b9-tls-certs\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.624525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.624473 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-model-cache\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.624567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.624534 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-tmp-dir\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.624608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.624577 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-home\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.626636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.626612 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-dshm\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.626920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.626903 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8112df-d5d4-41e4-adb2-560640e069b9-tls-certs\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.672507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.664566 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzjk\" (UniqueName: \"kubernetes.io/projected/8e8112df-d5d4-41e4-adb2-560640e069b9-kube-api-access-szzjk\") pod \"distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:31.783276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:31.783180 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:32.096725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:32.096632 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q"] Oct 30 06:10:32.100953 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:10:32.100921 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e8112df_d5d4_41e4_adb2_560640e069b9.slice/crio-2723a1483f25293cf9daccbb1e7d235085bdf882a43941b95c13f1d76ffa05e9 WatchSource:0}: Error finding container 2723a1483f25293cf9daccbb1e7d235085bdf882a43941b95c13f1d76ffa05e9: Status 404 returned error can't find the container with id 2723a1483f25293cf9daccbb1e7d235085bdf882a43941b95c13f1d76ffa05e9 Oct 30 06:10:32.117026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:32.117006 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c68d683-85ff-4411-bf67-d0f99a0c8d41" path="/var/lib/kubelet/pods/1c68d683-85ff-4411-bf67-d0f99a0c8d41/volumes" Oct 30 06:10:32.117525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:32.117510 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4973ef-1d32-41f2-bc92-286112948b10" path="/var/lib/kubelet/pods/3a4973ef-1d32-41f2-bc92-286112948b10/volumes" Oct 30 06:10:32.254343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:32.254290 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" event={"ID":"8e8112df-d5d4-41e4-adb2-560640e069b9","Type":"ContainerStarted","Data":"cc2bb5da0edfd84c93be543edacd0e19d33838512803360af7b59fb0cce2045a"} Oct 30 06:10:32.255464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:32.255413 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" event={"ID":"8e8112df-d5d4-41e4-adb2-560640e069b9","Type":"ContainerStarted","Data":"2723a1483f25293cf9daccbb1e7d235085bdf882a43941b95c13f1d76ffa05e9"} Oct 30 06:10:32.308412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:32.308355 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podStartSLOduration=1.308287542 podStartE2EDuration="1.308287542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:32.307687312 +0000 UTC m=+1046.303644323" watchObservedRunningTime="2026-10-30 06:10:32.308287542 +0000 UTC m=+1046.304244552" Oct 30 06:10:32.784034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:32.783994 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:36.322340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.322294 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc"] Oct 30 06:10:36.326280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.326262 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:36.344115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.344069 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:10:36.344115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.344080 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:10:36.347681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.347660 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc"] Oct 30 06:10:36.355461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.355431 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:10:36.355582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.355495 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:10:36.399120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.399091 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfsxl\" (UniqueName: \"kubernetes.io/projected/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-kube-api-access-vfsxl\") pod \"maas-api-revoke-sub-4de5a995-rnbmc\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:36.399272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.399140 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de5a995-rnbmc\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:36.399272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.399236 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de5a995-rnbmc\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:36.500451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.500411 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de5a995-rnbmc\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:36.500654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.500540 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfsxl\" (UniqueName: \"kubernetes.io/projected/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-kube-api-access-vfsxl\") pod \"maas-api-revoke-sub-4de5a995-rnbmc\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:36.500654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.500564 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de5a995-rnbmc\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:36.501211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.501190 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de5a995-rnbmc\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:36.503072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.503049 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de5a995-rnbmc\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:36.524825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.524795 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfsxl\" (UniqueName: \"kubernetes.io/projected/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-kube-api-access-vfsxl\") pod \"maas-api-revoke-sub-4de5a995-rnbmc\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:36.640316 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.640284 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:36.951871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:36.951839 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc"] Oct 30 06:10:36.954370 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:10:36.954345 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ccc380_344b_42c3_a5ca_d31b5c26cbd8.slice/crio-4b8939b6689d4630a2837922d96e35dcf512ed10f7f8280320108b2dc4e87200 WatchSource:0}: Error finding container 4b8939b6689d4630a2837922d96e35dcf512ed10f7f8280320108b2dc4e87200: Status 404 returned error can't find the container with id 4b8939b6689d4630a2837922d96e35dcf512ed10f7f8280320108b2dc4e87200 Oct 30 06:10:37.293714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:37.293675 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" event={"ID":"97ccc380-344b-42c3-a5ca-d31b5c26cbd8","Type":"ContainerStarted","Data":"d009592cb5659b30e4bd7ac2cbc48888a8d884cd91037c939328ff2b9649c306"} Oct 30 06:10:37.293889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:37.293736 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" event={"ID":"97ccc380-344b-42c3-a5ca-d31b5c26cbd8","Type":"ContainerStarted","Data":"4b8939b6689d4630a2837922d96e35dcf512ed10f7f8280320108b2dc4e87200"} Oct 30 06:10:37.336711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:37.336660 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" podStartSLOduration=1.336643434 podStartE2EDuration="1.336643434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:37.335745679 +0000 UTC m=+1051.331702689" watchObservedRunningTime="2026-10-30 06:10:37.336643434 +0000 UTC m=+1051.332600449" Oct 30 06:10:38.266183 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:10:38.266092 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:10:38.303684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:38.303647 5001 generic.go:356] "Generic (PLEG): container finished" podID="97ccc380-344b-42c3-a5ca-d31b5c26cbd8" containerID="d009592cb5659b30e4bd7ac2cbc48888a8d884cd91037c939328ff2b9649c306" exitCode=0 Oct 30 06:10:38.303867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:38.303694 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" event={"ID":"97ccc380-344b-42c3-a5ca-d31b5c26cbd8","Type":"ContainerDied","Data":"d009592cb5659b30e4bd7ac2cbc48888a8d884cd91037c939328ff2b9649c306"} Oct 30 06:10:39.463782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:39.463752 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:39.536793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:39.536764 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-openshift-ca\") pod \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " Oct 30 06:10:39.536956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:39.536824 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-xks-ca\") pod \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " Oct 30 06:10:39.537011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:39.536994 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-kube-api-access-vfsxl\" (UniqueName: \"kubernetes.io/projected/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-kube-api-access-vfsxl\") pod \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\" (UID: \"97ccc380-344b-42c3-a5ca-d31b5c26cbd8\") " Oct 30 06:10:39.537227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:39.537199 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-openshift-ca" pod "97ccc380-344b-42c3-a5ca-d31b5c26cbd8" (UID: "97ccc380-344b-42c3-a5ca-d31b5c26cbd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:39.537477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:39.537454 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:39.539149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:39.539118 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-xks-ca" pod "97ccc380-344b-42c3-a5ca-d31b5c26cbd8" (UID: "97ccc380-344b-42c3-a5ca-d31b5c26cbd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:39.539362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:39.539181 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-kube-api-access-vfsxl" pod "97ccc380-344b-42c3-a5ca-d31b5c26cbd8" (UID: "97ccc380-344b-42c3-a5ca-d31b5c26cbd8"). InnerVolumeSpecName "kube-api-access-vfsxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:39.638939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:39.638898 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfsxl\" (UniqueName: \"kubernetes.io/projected/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-kube-api-access-vfsxl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:39.638939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:39.638939 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ccc380-344b-42c3-a5ca-d31b5c26cbd8-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:40.145740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.145681 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" podUID="672125e5-94a5-4e3c-8cde-644966e57e0f" containerName="main" containerID="cri-o://3849c40becf4bbdc132586b0d4e8c1cb838beb6e2c3e9b687f3f8d2d2baa57f6" gracePeriod=45 Oct 30 06:10:40.320618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.320056 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" Oct 30 06:10:40.320618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.320085 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc" event={"ID":"97ccc380-344b-42c3-a5ca-d31b5c26cbd8","Type":"ContainerDied","Data":"4b8939b6689d4630a2837922d96e35dcf512ed10f7f8280320108b2dc4e87200"} Oct 30 06:10:40.320618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.320144 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b8939b6689d4630a2837922d96e35dcf512ed10f7f8280320108b2dc4e87200" Oct 30 06:10:40.322847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.322822 5001 generic.go:356] "Generic (PLEG): container finished" podID="672125e5-94a5-4e3c-8cde-644966e57e0f" containerID="3849c40becf4bbdc132586b0d4e8c1cb838beb6e2c3e9b687f3f8d2d2baa57f6" exitCode=0 Oct 30 06:10:40.323050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.322897 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" event={"ID":"672125e5-94a5-4e3c-8cde-644966e57e0f","Type":"ContainerDied","Data":"3849c40becf4bbdc132586b0d4e8c1cb838beb6e2c3e9b687f3f8d2d2baa57f6"} Oct 30 06:10:40.411442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.411415 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:10:40.548389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.548355 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672125e5-94a5-4e3c-8cde-644966e57e0f-kube-api-access-pxc4k\" (UniqueName: \"kubernetes.io/projected/672125e5-94a5-4e3c-8cde-644966e57e0f-kube-api-access-pxc4k\") pod \"672125e5-94a5-4e3c-8cde-644966e57e0f\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " Oct 30 06:10:40.548770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.548429 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-home\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-home\") pod \"672125e5-94a5-4e3c-8cde-644966e57e0f\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " Oct 30 06:10:40.548770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.548451 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/672125e5-94a5-4e3c-8cde-644966e57e0f-tls-certs\" (UniqueName: \"kubernetes.io/secret/672125e5-94a5-4e3c-8cde-644966e57e0f-tls-certs\") pod \"672125e5-94a5-4e3c-8cde-644966e57e0f\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " Oct 30 06:10:40.548770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.548494 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-tmp-dir\") pod \"672125e5-94a5-4e3c-8cde-644966e57e0f\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " Oct 30 06:10:40.548770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.548549 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-model-cache\") pod \"672125e5-94a5-4e3c-8cde-644966e57e0f\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " Oct 30 06:10:40.548770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.548642 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-dshm\") pod \"672125e5-94a5-4e3c-8cde-644966e57e0f\" (UID: \"672125e5-94a5-4e3c-8cde-644966e57e0f\") " Oct 30 06:10:40.548770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.548740 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-home" pod "672125e5-94a5-4e3c-8cde-644966e57e0f" (UID: "672125e5-94a5-4e3c-8cde-644966e57e0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:40.549104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.548786 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-tmp-dir" pod "672125e5-94a5-4e3c-8cde-644966e57e0f" (UID: "672125e5-94a5-4e3c-8cde-644966e57e0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:40.549104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.548869 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-model-cache" pod "672125e5-94a5-4e3c-8cde-644966e57e0f" (UID: "672125e5-94a5-4e3c-8cde-644966e57e0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:40.549104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.549048 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:40.549104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.549063 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:40.549104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.549080 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:40.550824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.550797 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672125e5-94a5-4e3c-8cde-644966e57e0f-kube-api-access-pxc4k" pod "672125e5-94a5-4e3c-8cde-644966e57e0f" (UID: "672125e5-94a5-4e3c-8cde-644966e57e0f"). InnerVolumeSpecName "kube-api-access-pxc4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:40.550824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.550801 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-dshm" pod "672125e5-94a5-4e3c-8cde-644966e57e0f" (UID: "672125e5-94a5-4e3c-8cde-644966e57e0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:40.550974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.550834 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/672125e5-94a5-4e3c-8cde-644966e57e0f-tls-certs" pod "672125e5-94a5-4e3c-8cde-644966e57e0f" (UID: "672125e5-94a5-4e3c-8cde-644966e57e0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:40.650027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.649990 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxc4k\" (UniqueName: \"kubernetes.io/projected/672125e5-94a5-4e3c-8cde-644966e57e0f-kube-api-access-pxc4k\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:40.650027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.650020 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672125e5-94a5-4e3c-8cde-644966e57e0f-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:40.650027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:40.650032 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672125e5-94a5-4e3c-8cde-644966e57e0f-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:41.330707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:41.330667 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" event={"ID":"672125e5-94a5-4e3c-8cde-644966e57e0f","Type":"ContainerDied","Data":"7afb1871122e44e6200afefc7560f51e2286be6ac3282ab25e9f1ed3869819f6"} Oct 30 06:10:41.330904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:41.330731 5001 scope.go:122] "RemoveContainer" containerID="3849c40becf4bbdc132586b0d4e8c1cb838beb6e2c3e9b687f3f8d2d2baa57f6" Oct 30 06:10:41.330904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:41.330741 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw" Oct 30 06:10:41.393788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:41.393605 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw"] Oct 30 06:10:41.425607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:41.425572 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-b-ce7e342c-kserve-6b7bb5f7f7-w8xbw"] Oct 30 06:10:41.784079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:41.784043 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:41.796972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:41.796946 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:42.116817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:42.116786 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672125e5-94a5-4e3c-8cde-644966e57e0f" path="/var/lib/kubelet/pods/672125e5-94a5-4e3c-8cde-644966e57e0f/volumes" Oct 30 06:10:42.348959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:42.348932 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:10:43.557377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:43.557338 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:10:48.081780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.081745 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4"] Oct 30 06:10:48.082244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.082232 5001 cpu_manager.go:414] "containerMap: removing container" podUID="97ccc380-344b-42c3-a5ca-d31b5c26cbd8" containerName="revoke-keys" Oct 30 06:10:48.082299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.082246 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ccc380-344b-42c3-a5ca-d31b5c26cbd8" containerName="revoke-keys" Oct 30 06:10:48.082299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.082260 5001 cpu_manager.go:414] "containerMap: removing container" podUID="672125e5-94a5-4e3c-8cde-644966e57e0f" containerName="main" Oct 30 06:10:48.082299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.082265 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672125e5-94a5-4e3c-8cde-644966e57e0f" containerName="main" Oct 30 06:10:48.082440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.082364 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ccc380-344b-42c3-a5ca-d31b5c26cbd8" containerName="revoke-keys" Oct 30 06:10:48.082440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.082374 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="672125e5-94a5-4e3c-8cde-644966e57e0f" containerName="main" Oct 30 06:10:48.087138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.087118 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:48.104138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.104101 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:10:48.104284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.104180 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:10:48.105862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.105835 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4"] Oct 30 06:10:48.115270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.115208 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:10:48.116956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.116927 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:10:48.242567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.242535 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-982aaa4c-jhzm4\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:48.242767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.242663 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twvqt\" (UniqueName: \"kubernetes.io/projected/388b8fa3-e252-411f-8e88-6a12feb86d69-kube-api-access-twvqt\") pod \"maas-api-revoke-sub-982aaa4c-jhzm4\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:48.242767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.242740 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-982aaa4c-jhzm4\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:48.343750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.343659 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twvqt\" (UniqueName: \"kubernetes.io/projected/388b8fa3-e252-411f-8e88-6a12feb86d69-kube-api-access-twvqt\") pod \"maas-api-revoke-sub-982aaa4c-jhzm4\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:48.343750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.343710 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-982aaa4c-jhzm4\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:48.343994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.343768 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-982aaa4c-jhzm4\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:48.344481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.344461 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-982aaa4c-jhzm4\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:48.346251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.346232 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-982aaa4c-jhzm4\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:48.370873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.370848 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twvqt\" (UniqueName: \"kubernetes.io/projected/388b8fa3-e252-411f-8e88-6a12feb86d69-kube-api-access-twvqt\") pod \"maas-api-revoke-sub-982aaa4c-jhzm4\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:48.402054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.402023 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:48.708106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:48.708071 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4"] Oct 30 06:10:48.709158 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:10:48.709134 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388b8fa3_e252_411f_8e88_6a12feb86d69.slice/crio-9f73ef476d782093b5886245e2859cc9939f279441373416ea461072f68715a8 WatchSource:0}: Error finding container 9f73ef476d782093b5886245e2859cc9939f279441373416ea461072f68715a8: Status 404 returned error can't find the container with id 9f73ef476d782093b5886245e2859cc9939f279441373416ea461072f68715a8 Oct 30 06:10:49.438910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.438870 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" event={"ID":"388b8fa3-e252-411f-8e88-6a12feb86d69","Type":"ContainerStarted","Data":"95d1c71ded133ce53dd9ae12fbf589dfe29b914f645b49b2ccd3d2f0a6020673"} Oct 30 06:10:49.439411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.438937 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" event={"ID":"388b8fa3-e252-411f-8e88-6a12feb86d69","Type":"ContainerStarted","Data":"9f73ef476d782093b5886245e2859cc9939f279441373416ea461072f68715a8"} Oct 30 06:10:49.485420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.485364 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" podStartSLOduration=1.485348715 podStartE2EDuration="1.485348715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:49.481271979 +0000 UTC m=+1063.477229000" watchObservedRunningTime="2026-10-30 06:10:49.485348715 +0000 UTC m=+1063.481305721" Oct 30 06:10:49.790785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.790691 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2"] Oct 30 06:10:49.794824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.794802 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:49.828281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.828249 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2"] Oct 30 06:10:49.961037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.961003 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:49.961208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.961113 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63eca24a-c41d-4d53-b691-98d71d929310-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:49.961208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.961143 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:49.961287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.961226 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63eca24a-c41d-4d53-b691-98d71d929310-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:49.961287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.961256 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qff6f\" (UniqueName: \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-kube-api-access-qff6f\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:49.961420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.961405 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:49.961479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.961468 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:49.961514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.961492 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:49.961560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:49.961544 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.062875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.062776 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.063058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.062875 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.063058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.062908 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.063058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.062945 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.063058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.062988 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.063058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.063052 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63eca24a-c41d-4d53-b691-98d71d929310-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.063369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.063082 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.063826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.063802 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63eca24a-c41d-4d53-b691-98d71d929310-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.064005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.063988 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qff6f\" (UniqueName: \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-kube-api-access-qff6f\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.064188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.063844 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.064339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.064043 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.070324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.064626 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.070324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.065112 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63eca24a-c41d-4d53-b691-98d71d929310-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.074135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.074105 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.074591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.074572 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63eca24a-c41d-4d53-b691-98d71d929310-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.075205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.075180 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.088349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.088296 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.088456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.088404 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qff6f\" (UniqueName: \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-kube-api-access-qff6f\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.109504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.109480 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:50.443129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.443080 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2"] Oct 30 06:10:50.447975 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:10:50.447944 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63eca24a_c41d_4d53_b691_98d71d929310.slice/crio-12f347f4d4cec1193b9600b159f30176b8202fdb43c77bbca6f5f7755f5d383e WatchSource:0}: Error finding container 12f347f4d4cec1193b9600b159f30176b8202fdb43c77bbca6f5f7755f5d383e: Status 404 returned error can't find the container with id 12f347f4d4cec1193b9600b159f30176b8202fdb43c77bbca6f5f7755f5d383e Oct 30 06:10:50.450924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.450894 5001 generic.go:356] "Generic (PLEG): container finished" podID="388b8fa3-e252-411f-8e88-6a12feb86d69" containerID="95d1c71ded133ce53dd9ae12fbf589dfe29b914f645b49b2ccd3d2f0a6020673" exitCode=0 Oct 30 06:10:50.451017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.450975 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" event={"ID":"388b8fa3-e252-411f-8e88-6a12feb86d69","Type":"ContainerDied","Data":"95d1c71ded133ce53dd9ae12fbf589dfe29b914f645b49b2ccd3d2f0a6020673"} Oct 30 06:10:50.451017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.450997 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:50.451111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.451096 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:50.451169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:50.451154 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:51.484570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.484531 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" event={"ID":"63eca24a-c41d-4d53-b691-98d71d929310","Type":"ContainerStarted","Data":"2352bf41ac2e3e5cf43c05d395a8ffe99ce9a2d2611e487abf914879e3627df3"} Oct 30 06:10:51.485086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.484591 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" event={"ID":"63eca24a-c41d-4d53-b691-98d71d929310","Type":"ContainerStarted","Data":"12f347f4d4cec1193b9600b159f30176b8202fdb43c77bbca6f5f7755f5d383e"} Oct 30 06:10:51.581344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.560929 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" podStartSLOduration=2.560907295 podStartE2EDuration="2.560907295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:51.547631468 +0000 UTC m=+1065.543588478" watchObservedRunningTime="2026-10-30 06:10:51.560907295 +0000 UTC m=+1065.556864307" Oct 30 06:10:51.698348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.696049 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:51.797083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.796698 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-xks-ca\") pod \"388b8fa3-e252-411f-8e88-6a12feb86d69\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " Oct 30 06:10:51.797083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.796746 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388b8fa3-e252-411f-8e88-6a12feb86d69-kube-api-access-twvqt\" (UniqueName: \"kubernetes.io/projected/388b8fa3-e252-411f-8e88-6a12feb86d69-kube-api-access-twvqt\") pod \"388b8fa3-e252-411f-8e88-6a12feb86d69\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " Oct 30 06:10:51.797083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.796768 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-openshift-ca\") pod \"388b8fa3-e252-411f-8e88-6a12feb86d69\" (UID: \"388b8fa3-e252-411f-8e88-6a12feb86d69\") " Oct 30 06:10:51.797384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.797362 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-openshift-ca" pod "388b8fa3-e252-411f-8e88-6a12feb86d69" (UID: "388b8fa3-e252-411f-8e88-6a12feb86d69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:51.801870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.801833 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388b8fa3-e252-411f-8e88-6a12feb86d69-kube-api-access-twvqt" pod "388b8fa3-e252-411f-8e88-6a12feb86d69" (UID: "388b8fa3-e252-411f-8e88-6a12feb86d69"). InnerVolumeSpecName "kube-api-access-twvqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:51.803419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.803395 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-xks-ca" pod "388b8fa3-e252-411f-8e88-6a12feb86d69" (UID: "388b8fa3-e252-411f-8e88-6a12feb86d69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:51.898231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.898197 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:51.898231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.898233 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twvqt\" (UniqueName: \"kubernetes.io/projected/388b8fa3-e252-411f-8e88-6a12feb86d69-kube-api-access-twvqt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:51.898455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.898250 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388b8fa3-e252-411f-8e88-6a12feb86d69-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:10:51.955334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.952029 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j"] Oct 30 06:10:51.978343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.969980 5001 cpu_manager.go:414] "containerMap: removing container" podUID="388b8fa3-e252-411f-8e88-6a12feb86d69" containerName="revoke-keys" Oct 30 06:10:51.978343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.970014 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388b8fa3-e252-411f-8e88-6a12feb86d69" containerName="revoke-keys" Oct 30 06:10:51.978343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.970148 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="388b8fa3-e252-411f-8e88-6a12feb86d69" containerName="revoke-keys" Oct 30 06:10:51.990754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.987454 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:51.990925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.990894 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j"] Oct 30 06:10:51.999209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.999178 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:51.999390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.999224 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:51.999390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.999251 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ed7d68-efcb-424b-8289-ad2c278768c2-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:51.999390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.999318 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vmb\" (UniqueName: \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-kube-api-access-d4vmb\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:51.999390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.999379 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:51.999661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.999404 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:51.999661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.999430 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:51.999661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.999504 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:51.999661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:51.999566 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.006089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.006059 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-eaee88-openshift-default-dockercfg-fmzvg\"" Oct 30 06:10:52.100514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.100412 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.100514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.100454 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.100759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.100542 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.100759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.100565 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.100759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.100589 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ed7d68-efcb-424b-8289-ad2c278768c2-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.100759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.100619 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vmb\" (UniqueName: \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-kube-api-access-d4vmb\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.100759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.100662 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.100759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.100686 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.100759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.100710 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.101121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.100992 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.101352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.101293 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.101352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.101345 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.101505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.101427 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.101728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.101692 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ed7d68-efcb-424b-8289-ad2c278768c2-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.103448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.103418 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.103604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.103588 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.111764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.111737 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 06:10:52.111933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.111824 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" podUID="63eca24a-c41d-4d53-b691-98d71d929310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 06:10:52.117985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.117969 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:52.127298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.127276 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.127463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.127444 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vmb\" (UniqueName: \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-kube-api-access-d4vmb\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.310518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.310476 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:52.497563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.497529 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" Oct 30 06:10:52.498024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.497603 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4" event={"ID":"388b8fa3-e252-411f-8e88-6a12feb86d69","Type":"ContainerDied","Data":"9f73ef476d782093b5886245e2859cc9939f279441373416ea461072f68715a8"} Oct 30 06:10:52.498024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.497654 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f73ef476d782093b5886245e2859cc9939f279441373416ea461072f68715a8" Oct 30 06:10:52.687807 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:10:52.687778 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ed7d68_efcb_424b_8289_ad2c278768c2.slice/crio-63814e3160f56700e4c588df4fa3caf8323c6174d205575612cbfff8d712d876 WatchSource:0}: Error finding container 63814e3160f56700e4c588df4fa3caf8323c6174d205575612cbfff8d712d876: Status 404 returned error can't find the container with id 63814e3160f56700e4c588df4fa3caf8323c6174d205575612cbfff8d712d876 Oct 30 06:10:52.688038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.688008 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j"] Oct 30 06:10:52.691350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.690551 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:52.691350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.690648 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:52.691350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:52.690718 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:53.109939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:53.109907 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 06:10:53.110117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:53.109957 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" podUID="63eca24a-c41d-4d53-b691-98d71d929310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 06:10:53.519675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:53.519597 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" event={"ID":"56ed7d68-efcb-424b-8289-ad2c278768c2","Type":"ContainerStarted","Data":"8664586e94dd95a6a6bec68f7173ef732a675683eff5112a4e4394f3089ed6fb"} Oct 30 06:10:53.520130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:53.520110 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" event={"ID":"56ed7d68-efcb-424b-8289-ad2c278768c2","Type":"ContainerStarted","Data":"63814e3160f56700e4c588df4fa3caf8323c6174d205575612cbfff8d712d876"} Oct 30 06:10:54.113801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:54.113762 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 06:10:54.114100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:54.114071 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" podUID="63eca24a-c41d-4d53-b691-98d71d929310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 06:10:54.312745 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:54.312512 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:10:54.315172 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:54.315144 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:10:54.315419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:54.315393 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 06:10:55.112264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:55.111770 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 06:10:55.112264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:55.111842 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" podUID="63eca24a-c41d-4d53-b691-98d71d929310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 06:10:55.311758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:55.311324 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:10:55.311758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:55.311386 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 06:10:56.111746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:56.111712 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 06:10:56.111960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:56.111768 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" podUID="63eca24a-c41d-4d53-b691-98d71d929310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 06:10:56.313772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:56.313735 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:10:56.314398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:56.314362 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 06:10:57.117729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:57.117694 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 06:10:57.117997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:57.117969 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" podUID="63eca24a-c41d-4d53-b691-98d71d929310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 06:10:57.313733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:57.313694 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:10:57.313985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:57.313955 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 06:10:58.321692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.321641 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:10:58.340546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.322249 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 06:10:58.507334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.504742 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" podStartSLOduration=7.504725426 podStartE2EDuration="7.504725426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:53.587672307 +0000 UTC m=+1067.583629318" watchObservedRunningTime="2026-10-30 06:10:58.504725426 +0000 UTC m=+1072.500682436" Oct 30 06:10:58.507334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.507129 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f"] Oct 30 06:10:58.511817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.511791 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:10:58.528039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.528011 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:10:58.540994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.540960 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:10:58.541152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.541050 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:10:58.545187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.544589 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:10:58.549549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.549519 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f"] Oct 30 06:10:58.612673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.611941 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfkjl\" (UniqueName: \"kubernetes.io/projected/f315c894-1fe5-410f-ba57-0e0d24feb024-kube-api-access-zfkjl\") pod \"maas-api-revoke-sub-e188c1ae-wbl5f\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:10:58.612673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.612036 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e188c1ae-wbl5f\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:10:58.612673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.612174 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e188c1ae-wbl5f\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:10:58.713697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.713659 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e188c1ae-wbl5f\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:10:58.713875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.713812 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e188c1ae-wbl5f\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:10:58.713927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.713883 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfkjl\" (UniqueName: \"kubernetes.io/projected/f315c894-1fe5-410f-ba57-0e0d24feb024-kube-api-access-zfkjl\") pod \"maas-api-revoke-sub-e188c1ae-wbl5f\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:10:58.714463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.714439 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e188c1ae-wbl5f\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:10:58.716367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.716352 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e188c1ae-wbl5f\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:10:58.738259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.738229 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfkjl\" (UniqueName: \"kubernetes.io/projected/f315c894-1fe5-410f-ba57-0e0d24feb024-kube-api-access-zfkjl\") pod \"maas-api-revoke-sub-e188c1ae-wbl5f\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:10:58.746601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.746563 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:58.747462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.747436 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:58.748334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.748291 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:10:58.826745 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:58.826490 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:10:59.209866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:59.209837 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f"] Oct 30 06:10:59.214757 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:10:59.212674 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf315c894_1fe5_410f_ba57_0e0d24feb024.slice/crio-c74f332af1c8553794e5eceeea15ddaf091a745bb454be04f0884e3f25990f65 WatchSource:0}: Error finding container c74f332af1c8553794e5eceeea15ddaf091a745bb454be04f0884e3f25990f65: Status 404 returned error can't find the container with id c74f332af1c8553794e5eceeea15ddaf091a745bb454be04f0884e3f25990f65 Oct 30 06:10:59.311580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:59.311546 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:10:59.311773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:59.311609 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 06:10:59.627023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:10:59.626987 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" event={"ID":"f315c894-1fe5-410f-ba57-0e0d24feb024","Type":"ContainerStarted","Data":"c74f332af1c8553794e5eceeea15ddaf091a745bb454be04f0884e3f25990f65"} Oct 30 06:11:00.564240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:00.564204 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:11:00.565495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:00.565470 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:11:00.566271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:00.566246 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:11:00.634949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:00.634858 5001 generic.go:356] "Generic (PLEG): container finished" podID="f315c894-1fe5-410f-ba57-0e0d24feb024" containerID="3686f016abf86b8ae5ae489f4b0eeb86a1f40ce42666711be66c35b0ff193bde" exitCode=0 Oct 30 06:11:00.635391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:00.634948 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" event={"ID":"f315c894-1fe5-410f-ba57-0e0d24feb024","Type":"ContainerDied","Data":"3686f016abf86b8ae5ae489f4b0eeb86a1f40ce42666711be66c35b0ff193bde"} Oct 30 06:11:01.830701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:01.830674 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:11:01.990136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:01.990045 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f315c894-1fe5-410f-ba57-0e0d24feb024-kube-api-access-zfkjl\" (UniqueName: \"kubernetes.io/projected/f315c894-1fe5-410f-ba57-0e0d24feb024-kube-api-access-zfkjl\") pod \"f315c894-1fe5-410f-ba57-0e0d24feb024\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " Oct 30 06:11:01.990136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:01.990092 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-xks-ca\") pod \"f315c894-1fe5-410f-ba57-0e0d24feb024\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " Oct 30 06:11:01.990406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:01.990203 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-openshift-ca\") pod \"f315c894-1fe5-410f-ba57-0e0d24feb024\" (UID: \"f315c894-1fe5-410f-ba57-0e0d24feb024\") " Oct 30 06:11:01.990525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:01.990498 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-openshift-ca" pod "f315c894-1fe5-410f-ba57-0e0d24feb024" (UID: "f315c894-1fe5-410f-ba57-0e0d24feb024"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:01.992377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:01.992353 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f315c894-1fe5-410f-ba57-0e0d24feb024-kube-api-access-zfkjl" pod "f315c894-1fe5-410f-ba57-0e0d24feb024" (UID: "f315c894-1fe5-410f-ba57-0e0d24feb024"). InnerVolumeSpecName "kube-api-access-zfkjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:01.992452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:01.992432 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-xks-ca" pod "f315c894-1fe5-410f-ba57-0e0d24feb024" (UID: "f315c894-1fe5-410f-ba57-0e0d24feb024"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:02.091507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.091465 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:02.091507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.091502 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfkjl\" (UniqueName: \"kubernetes.io/projected/f315c894-1fe5-410f-ba57-0e0d24feb024-kube-api-access-zfkjl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:02.091725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.091522 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f315c894-1fe5-410f-ba57-0e0d24feb024-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:02.410519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.410471 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2"] Oct 30 06:11:02.411043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.411022 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f315c894-1fe5-410f-ba57-0e0d24feb024" containerName="revoke-keys" Oct 30 06:11:02.411043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.411041 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f315c894-1fe5-410f-ba57-0e0d24feb024" containerName="revoke-keys" Oct 30 06:11:02.411216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.411173 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f315c894-1fe5-410f-ba57-0e0d24feb024" containerName="revoke-keys" Oct 30 06:11:02.424741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.424712 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:02.435569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.435539 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2"] Oct 30 06:11:02.596888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.596834 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7305ea32-jrjp2\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:02.597095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.597071 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7305ea32-jrjp2\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:02.597429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.597241 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frd8j\" (UniqueName: \"kubernetes.io/projected/24c8170d-bb38-41a4-be87-2377cfdefb9d-kube-api-access-frd8j\") pod \"maas-api-revoke-sub-7305ea32-jrjp2\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:02.649493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.649460 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" Oct 30 06:11:02.649680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.649462 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f" event={"ID":"f315c894-1fe5-410f-ba57-0e0d24feb024","Type":"ContainerDied","Data":"c74f332af1c8553794e5eceeea15ddaf091a745bb454be04f0884e3f25990f65"} Oct 30 06:11:02.649680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.649586 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c74f332af1c8553794e5eceeea15ddaf091a745bb454be04f0884e3f25990f65" Oct 30 06:11:02.698561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.698475 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7305ea32-jrjp2\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:02.698734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.698587 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frd8j\" (UniqueName: \"kubernetes.io/projected/24c8170d-bb38-41a4-be87-2377cfdefb9d-kube-api-access-frd8j\") pod \"maas-api-revoke-sub-7305ea32-jrjp2\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:02.698734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.698621 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7305ea32-jrjp2\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:02.699352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.699270 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7305ea32-jrjp2\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:02.701037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.701013 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7305ea32-jrjp2\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:02.723841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.723819 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frd8j\" (UniqueName: \"kubernetes.io/projected/24c8170d-bb38-41a4-be87-2377cfdefb9d-kube-api-access-frd8j\") pod \"maas-api-revoke-sub-7305ea32-jrjp2\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:02.741573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:02.741548 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:03.047931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:03.047896 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2"] Oct 30 06:11:03.049030 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:03.049005 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c8170d_bb38_41a4_be87_2377cfdefb9d.slice/crio-42225a978818ba37dbea99e4d14f9cb22caf8b2710facf83bde4c282a18e29f7 WatchSource:0}: Error finding container 42225a978818ba37dbea99e4d14f9cb22caf8b2710facf83bde4c282a18e29f7: Status 404 returned error can't find the container with id 42225a978818ba37dbea99e4d14f9cb22caf8b2710facf83bde4c282a18e29f7 Oct 30 06:11:03.659013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:03.658919 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" event={"ID":"24c8170d-bb38-41a4-be87-2377cfdefb9d","Type":"ContainerStarted","Data":"dc0682a31533419b47aea92100c3fd5a9aedb7d8cc900258c237207a56821b9d"} Oct 30 06:11:03.659013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:03.658982 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" event={"ID":"24c8170d-bb38-41a4-be87-2377cfdefb9d","Type":"ContainerStarted","Data":"42225a978818ba37dbea99e4d14f9cb22caf8b2710facf83bde4c282a18e29f7"} Oct 30 06:11:03.704459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:03.704404 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" podStartSLOduration=1.704389105 podStartE2EDuration="1.704389105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:03.70340771 +0000 UTC m=+1077.699364719" watchObservedRunningTime="2026-10-30 06:11:03.704389105 +0000 UTC m=+1077.700346115" Oct 30 06:11:04.406387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.406354 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc"] Oct 30 06:11:04.410504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.410485 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.434953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.434918 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc"] Oct 30 06:11:04.528999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.528966 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24b04da-c878-487b-8805-18e48cfc7f37-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.529190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.529059 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.529190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.529081 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.529190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.529106 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4wb\" (UniqueName: \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-kube-api-access-db4wb\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.529190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.529137 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.529190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.529157 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.529190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.529173 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.529449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.529221 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.529449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.529243 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24b04da-c878-487b-8805-18e48cfc7f37-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.630243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630209 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.630243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630242 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.630518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630259 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4wb\" (UniqueName: \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-kube-api-access-db4wb\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.630518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630287 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.630518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630329 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.630518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630359 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.630518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630440 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.630518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630474 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24b04da-c878-487b-8805-18e48cfc7f37-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.630825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630535 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24b04da-c878-487b-8805-18e48cfc7f37-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.630825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630725 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.630973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630912 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.631191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630967 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.631191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.630992 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.631296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.631230 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24b04da-c878-487b-8805-18e48cfc7f37-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.632823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.632795 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.633199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.633182 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24b04da-c878-487b-8805-18e48cfc7f37-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.654650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.654618 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.654856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.654829 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4wb\" (UniqueName: \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-kube-api-access-db4wb\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:04.668934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.668853 5001 generic.go:356] "Generic (PLEG): container finished" podID="24c8170d-bb38-41a4-be87-2377cfdefb9d" containerID="dc0682a31533419b47aea92100c3fd5a9aedb7d8cc900258c237207a56821b9d" exitCode=0 Oct 30 06:11:04.669092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.669078 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" event={"ID":"24c8170d-bb38-41a4-be87-2377cfdefb9d","Type":"ContainerDied","Data":"dc0682a31533419b47aea92100c3fd5a9aedb7d8cc900258c237207a56821b9d"} Oct 30 06:11:04.723962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:04.723933 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:05.062841 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:05.062812 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24b04da_c878_487b_8805_18e48cfc7f37.slice/crio-affccdb40031ee1d5352ba3364f115c24dc05ebdbe694e8e86c172c7911ddd92 WatchSource:0}: Error finding container affccdb40031ee1d5352ba3364f115c24dc05ebdbe694e8e86c172c7911ddd92: Status 404 returned error can't find the container with id affccdb40031ee1d5352ba3364f115c24dc05ebdbe694e8e86c172c7911ddd92 Oct 30 06:11:05.063105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:05.063083 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc"] Oct 30 06:11:05.065286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:05.065256 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:05.065409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:05.065339 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:05.065409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:05.065372 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:05.685376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:05.685338 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" event={"ID":"e24b04da-c878-487b-8805-18e48cfc7f37","Type":"ContainerStarted","Data":"877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94"} Oct 30 06:11:05.685812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:05.685420 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" event={"ID":"e24b04da-c878-487b-8805-18e48cfc7f37","Type":"ContainerStarted","Data":"affccdb40031ee1d5352ba3364f115c24dc05ebdbe694e8e86c172c7911ddd92"} Oct 30 06:11:05.725224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:05.725097 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:05.789294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:05.789219 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" podStartSLOduration=1.789197278 podStartE2EDuration="1.789197278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:05.744380577 +0000 UTC m=+1079.740337588" watchObservedRunningTime="2026-10-30 06:11:05.789197278 +0000 UTC m=+1079.785154293" Oct 30 06:11:05.899328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:05.897221 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:06.047838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.047751 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-openshift-ca\") pod \"24c8170d-bb38-41a4-be87-2377cfdefb9d\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " Oct 30 06:11:06.047838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.047799 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c8170d-bb38-41a4-be87-2377cfdefb9d-kube-api-access-frd8j\" (UniqueName: \"kubernetes.io/projected/24c8170d-bb38-41a4-be87-2377cfdefb9d-kube-api-access-frd8j\") pod \"24c8170d-bb38-41a4-be87-2377cfdefb9d\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " Oct 30 06:11:06.048076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.047856 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-xks-ca\") pod \"24c8170d-bb38-41a4-be87-2377cfdefb9d\" (UID: \"24c8170d-bb38-41a4-be87-2377cfdefb9d\") " Oct 30 06:11:06.048210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.048180 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-openshift-ca" pod "24c8170d-bb38-41a4-be87-2377cfdefb9d" (UID: "24c8170d-bb38-41a4-be87-2377cfdefb9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:06.050215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.050182 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c8170d-bb38-41a4-be87-2377cfdefb9d-kube-api-access-frd8j" pod "24c8170d-bb38-41a4-be87-2377cfdefb9d" (UID: "24c8170d-bb38-41a4-be87-2377cfdefb9d"). InnerVolumeSpecName "kube-api-access-frd8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:06.050376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.050238 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-xks-ca" pod "24c8170d-bb38-41a4-be87-2377cfdefb9d" (UID: "24c8170d-bb38-41a4-be87-2377cfdefb9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:06.149776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.149748 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:06.149776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.149774 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c8170d-bb38-41a4-be87-2377cfdefb9d-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:06.150026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.149786 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frd8j\" (UniqueName: \"kubernetes.io/projected/24c8170d-bb38-41a4-be87-2377cfdefb9d-kube-api-access-frd8j\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:06.694175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.694145 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" Oct 30 06:11:06.694175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.694153 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2" event={"ID":"24c8170d-bb38-41a4-be87-2377cfdefb9d","Type":"ContainerDied","Data":"42225a978818ba37dbea99e4d14f9cb22caf8b2710facf83bde4c282a18e29f7"} Oct 30 06:11:06.694755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.694209 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42225a978818ba37dbea99e4d14f9cb22caf8b2710facf83bde4c282a18e29f7" Oct 30 06:11:06.726288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.726258 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 06:11:06.726473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:06.726322 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 06:11:07.725103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:07.725070 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 06:11:07.725510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:07.725120 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 06:11:08.351386 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:11:08.349464 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:11:08.724996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:08.724952 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 06:11:08.725185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:08.725018 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 06:11:09.724945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:09.724912 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 06:11:09.725123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:09.724976 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 06:11:10.724999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:10.724969 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 06:11:10.725419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:10.725023 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 06:11:11.638551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.638512 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6"] Oct 30 06:11:11.639409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.639387 5001 cpu_manager.go:414] "containerMap: removing container" podUID="24c8170d-bb38-41a4-be87-2377cfdefb9d" containerName="revoke-keys" Oct 30 06:11:11.639530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.639411 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c8170d-bb38-41a4-be87-2377cfdefb9d" containerName="revoke-keys" Oct 30 06:11:11.639592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.639550 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c8170d-bb38-41a4-be87-2377cfdefb9d" containerName="revoke-keys" Oct 30 06:11:11.644979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.644952 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:11.658977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.658174 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:11:11.662448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.662247 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:11:11.662448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.662405 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:11:11.662843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.662573 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:11.664616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.663989 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:11:11.687561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.687505 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6"] Oct 30 06:11:11.725128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.725090 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 06:11:11.725678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.725193 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 06:11:11.745466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.745420 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70652026-m5gb6\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:11.745466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.745467 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70652026-m5gb6\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:11.745725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.745585 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv2kc\" (UniqueName: \"kubernetes.io/projected/a4094479-4ea6-4004-9ae7-06ab553974c3-kube-api-access-xv2kc\") pod \"maas-api-revoke-sub-70652026-m5gb6\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:11.846968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.846930 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70652026-m5gb6\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:11.847178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.846978 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70652026-m5gb6\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:11.847178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.847048 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv2kc\" (UniqueName: \"kubernetes.io/projected/a4094479-4ea6-4004-9ae7-06ab553974c3-kube-api-access-xv2kc\") pod \"maas-api-revoke-sub-70652026-m5gb6\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:11.847809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.847782 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70652026-m5gb6\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:11.849914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.849887 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70652026-m5gb6\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:11.874009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.873966 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv2kc\" (UniqueName: \"kubernetes.io/projected/a4094479-4ea6-4004-9ae7-06ab553974c3-kube-api-access-xv2kc\") pod \"maas-api-revoke-sub-70652026-m5gb6\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:11.990238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:11.990145 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:12.410684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:12.410598 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6"] Oct 30 06:11:12.410983 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:12.410939 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4094479_4ea6_4004_9ae7_06ab553974c3.slice/crio-adb31f11118c792ba3250758710aa7c9a916300cfad030e9abb98267e0042bca WatchSource:0}: Error finding container adb31f11118c792ba3250758710aa7c9a916300cfad030e9abb98267e0042bca: Status 404 returned error can't find the container with id adb31f11118c792ba3250758710aa7c9a916300cfad030e9abb98267e0042bca Oct 30 06:11:12.749183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:12.748766 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" event={"ID":"a4094479-4ea6-4004-9ae7-06ab553974c3","Type":"ContainerStarted","Data":"adb31f11118c792ba3250758710aa7c9a916300cfad030e9abb98267e0042bca"} Oct 30 06:11:12.773100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:12.773066 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:12.774598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:12.774568 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:12.788114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:12.778706 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:11:13.800331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:13.800146 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" event={"ID":"a4094479-4ea6-4004-9ae7-06ab553974c3","Type":"ContainerStarted","Data":"c3662750d72d61bce7a02c023b918e8e1715e6a4f0a652530f654b72ee63c3ac"} Oct 30 06:11:13.886332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:13.871799 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" podStartSLOduration=2.871776305 podStartE2EDuration="2.871776305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:13.863639134 +0000 UTC m=+1087.859596206" watchObservedRunningTime="2026-10-30 06:11:13.871776305 +0000 UTC m=+1087.867733316" Oct 30 06:11:14.820412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:14.820363 5001 generic.go:356] "Generic (PLEG): container finished" podID="a4094479-4ea6-4004-9ae7-06ab553974c3" containerID="c3662750d72d61bce7a02c023b918e8e1715e6a4f0a652530f654b72ee63c3ac" exitCode=0 Oct 30 06:11:14.826156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:14.826129 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" event={"ID":"a4094479-4ea6-4004-9ae7-06ab553974c3","Type":"ContainerDied","Data":"c3662750d72d61bce7a02c023b918e8e1715e6a4f0a652530f654b72ee63c3ac"} Oct 30 06:11:16.007835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.007809 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:16.043793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.043764 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-openshift-ca\") pod \"a4094479-4ea6-4004-9ae7-06ab553974c3\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " Oct 30 06:11:16.044008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.043943 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4094479-4ea6-4004-9ae7-06ab553974c3-kube-api-access-xv2kc\" (UniqueName: \"kubernetes.io/projected/a4094479-4ea6-4004-9ae7-06ab553974c3-kube-api-access-xv2kc\") pod \"a4094479-4ea6-4004-9ae7-06ab553974c3\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " Oct 30 06:11:16.044008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.043997 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-xks-ca\") pod \"a4094479-4ea6-4004-9ae7-06ab553974c3\" (UID: \"a4094479-4ea6-4004-9ae7-06ab553974c3\") " Oct 30 06:11:16.044216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.044194 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-openshift-ca" pod "a4094479-4ea6-4004-9ae7-06ab553974c3" (UID: "a4094479-4ea6-4004-9ae7-06ab553974c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:16.044499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.044480 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:16.046359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.046324 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-xks-ca" pod "a4094479-4ea6-4004-9ae7-06ab553974c3" (UID: "a4094479-4ea6-4004-9ae7-06ab553974c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:16.046463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.046402 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4094479-4ea6-4004-9ae7-06ab553974c3-kube-api-access-xv2kc" pod "a4094479-4ea6-4004-9ae7-06ab553974c3" (UID: "a4094479-4ea6-4004-9ae7-06ab553974c3"). InnerVolumeSpecName "kube-api-access-xv2kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:16.146199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.146172 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv2kc\" (UniqueName: \"kubernetes.io/projected/a4094479-4ea6-4004-9ae7-06ab553974c3-kube-api-access-xv2kc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:16.146199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.146199 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4094479-4ea6-4004-9ae7-06ab553974c3-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:16.837378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.837343 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" Oct 30 06:11:16.837378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.837356 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6" event={"ID":"a4094479-4ea6-4004-9ae7-06ab553974c3","Type":"ContainerDied","Data":"adb31f11118c792ba3250758710aa7c9a916300cfad030e9abb98267e0042bca"} Oct 30 06:11:16.837602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:16.837440 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adb31f11118c792ba3250758710aa7c9a916300cfad030e9abb98267e0042bca" Oct 30 06:11:17.486107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.486071 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k"] Oct 30 06:11:17.486610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.486599 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a4094479-4ea6-4004-9ae7-06ab553974c3" containerName="revoke-keys" Oct 30 06:11:17.486698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.486611 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4094479-4ea6-4004-9ae7-06ab553974c3" containerName="revoke-keys" Oct 30 06:11:17.486754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.486726 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4094479-4ea6-4004-9ae7-06ab553974c3" containerName="revoke-keys" Oct 30 06:11:17.490357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.490337 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:17.511520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.511495 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k"] Oct 30 06:11:17.520520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.520498 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:11:17.520520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.520498 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:11:17.522520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.522498 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:17.522636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.522596 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:11:17.562853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.562818 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12494765-b808-4020-af35-349e1b46fa46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b64b77-87p2k\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:17.563029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.562967 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztmpf\" (UniqueName: \"kubernetes.io/projected/12494765-b808-4020-af35-349e1b46fa46-kube-api-access-ztmpf\") pod \"maas-api-revoke-sub-54b64b77-87p2k\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:17.563029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.563002 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12494765-b808-4020-af35-349e1b46fa46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b64b77-87p2k\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:17.664015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.663978 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztmpf\" (UniqueName: \"kubernetes.io/projected/12494765-b808-4020-af35-349e1b46fa46-kube-api-access-ztmpf\") pod \"maas-api-revoke-sub-54b64b77-87p2k\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:17.664214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.664032 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12494765-b808-4020-af35-349e1b46fa46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b64b77-87p2k\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:17.664214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.664164 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12494765-b808-4020-af35-349e1b46fa46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b64b77-87p2k\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:17.664889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.664857 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12494765-b808-4020-af35-349e1b46fa46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b64b77-87p2k\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:17.666784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.666758 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12494765-b808-4020-af35-349e1b46fa46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b64b77-87p2k\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:17.688935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.688902 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztmpf\" (UniqueName: \"kubernetes.io/projected/12494765-b808-4020-af35-349e1b46fa46-kube-api-access-ztmpf\") pod \"maas-api-revoke-sub-54b64b77-87p2k\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:17.805623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:17.805540 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:18.117517 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:18.117452 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12494765_b808_4020_af35_349e1b46fa46.slice/crio-bfc74172094f2c2d838c0b7474c355bfb3248d76b9f15533294e47c798ec318c WatchSource:0}: Error finding container bfc74172094f2c2d838c0b7474c355bfb3248d76b9f15533294e47c798ec318c: Status 404 returned error can't find the container with id bfc74172094f2c2d838c0b7474c355bfb3248d76b9f15533294e47c798ec318c Oct 30 06:11:18.118056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:18.118037 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k"] Oct 30 06:11:18.855621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:18.855586 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" event={"ID":"12494765-b808-4020-af35-349e1b46fa46","Type":"ContainerStarted","Data":"045fde5882ad25385ee17b2f9e09e4fc34e2410ac4f6298c3e0058756ca64a47"} Oct 30 06:11:18.856135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:18.855646 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" event={"ID":"12494765-b808-4020-af35-349e1b46fa46","Type":"ContainerStarted","Data":"bfc74172094f2c2d838c0b7474c355bfb3248d76b9f15533294e47c798ec318c"} Oct 30 06:11:18.899837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:18.899782 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" podStartSLOduration=1.899768059 podStartE2EDuration="1.899768059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:18.897723043 +0000 UTC m=+1092.893680053" watchObservedRunningTime="2026-10-30 06:11:18.899768059 +0000 UTC m=+1092.895725068" Oct 30 06:11:19.863804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:19.863770 5001 generic.go:356] "Generic (PLEG): container finished" podID="12494765-b808-4020-af35-349e1b46fa46" containerID="045fde5882ad25385ee17b2f9e09e4fc34e2410ac4f6298c3e0058756ca64a47" exitCode=0 Oct 30 06:11:19.864283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:19.863977 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" event={"ID":"12494765-b808-4020-af35-349e1b46fa46","Type":"ContainerDied","Data":"045fde5882ad25385ee17b2f9e09e4fc34e2410ac4f6298c3e0058756ca64a47"} Oct 30 06:11:21.021998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.021970 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:21.106131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.106087 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12494765-b808-4020-af35-349e1b46fa46-kube-api-access-ztmpf\" (UniqueName: \"kubernetes.io/projected/12494765-b808-4020-af35-349e1b46fa46-kube-api-access-ztmpf\") pod \"12494765-b808-4020-af35-349e1b46fa46\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " Oct 30 06:11:21.106330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.106215 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12494765-b808-4020-af35-349e1b46fa46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12494765-b808-4020-af35-349e1b46fa46-maas-api-openshift-ca\") pod \"12494765-b808-4020-af35-349e1b46fa46\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " Oct 30 06:11:21.106417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.106329 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12494765-b808-4020-af35-349e1b46fa46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12494765-b808-4020-af35-349e1b46fa46-maas-api-xks-ca\") pod \"12494765-b808-4020-af35-349e1b46fa46\" (UID: \"12494765-b808-4020-af35-349e1b46fa46\") " Oct 30 06:11:21.109719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.109682 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12494765-b808-4020-af35-349e1b46fa46-maas-api-openshift-ca" pod "12494765-b808-4020-af35-349e1b46fa46" (UID: "12494765-b808-4020-af35-349e1b46fa46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:21.110355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.110327 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12494765-b808-4020-af35-349e1b46fa46-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:21.114638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.114590 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12494765-b808-4020-af35-349e1b46fa46-maas-api-xks-ca" pod "12494765-b808-4020-af35-349e1b46fa46" (UID: "12494765-b808-4020-af35-349e1b46fa46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:21.114877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.114852 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12494765-b808-4020-af35-349e1b46fa46-kube-api-access-ztmpf" pod "12494765-b808-4020-af35-349e1b46fa46" (UID: "12494765-b808-4020-af35-349e1b46fa46"). InnerVolumeSpecName "kube-api-access-ztmpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:21.211561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.211526 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztmpf\" (UniqueName: \"kubernetes.io/projected/12494765-b808-4020-af35-349e1b46fa46-kube-api-access-ztmpf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:21.211561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.211558 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12494765-b808-4020-af35-349e1b46fa46-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:21.879084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.879054 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" Oct 30 06:11:21.879259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.879088 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k" event={"ID":"12494765-b808-4020-af35-349e1b46fa46","Type":"ContainerDied","Data":"bfc74172094f2c2d838c0b7474c355bfb3248d76b9f15533294e47c798ec318c"} Oct 30 06:11:21.879259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:21.879178 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfc74172094f2c2d838c0b7474c355bfb3248d76b9f15533294e47c798ec318c" Oct 30 06:11:25.580762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.580714 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v"] Oct 30 06:11:25.581520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.581496 5001 cpu_manager.go:414] "containerMap: removing container" podUID="12494765-b808-4020-af35-349e1b46fa46" containerName="revoke-keys" Oct 30 06:11:25.581520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.581521 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12494765-b808-4020-af35-349e1b46fa46" containerName="revoke-keys" Oct 30 06:11:25.581739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.581725 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="12494765-b808-4020-af35-349e1b46fa46" containerName="revoke-keys" Oct 30 06:11:25.585674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.585644 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:25.618782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.618736 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v"] Oct 30 06:11:25.619150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.619131 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:25.619578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.619552 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:11:25.619947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.619926 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:11:25.620390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.620369 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:11:25.679720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.679685 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efed0ff1-n5m2v\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:25.680051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.680029 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efed0ff1-n5m2v\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:25.680169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.680156 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ksbk\" (UniqueName: \"kubernetes.io/projected/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-kube-api-access-6ksbk\") pod \"maas-api-revoke-sub-efed0ff1-n5m2v\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:25.783966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.783924 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ksbk\" (UniqueName: \"kubernetes.io/projected/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-kube-api-access-6ksbk\") pod \"maas-api-revoke-sub-efed0ff1-n5m2v\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:25.784763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.784738 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efed0ff1-n5m2v\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:25.790266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.790241 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efed0ff1-n5m2v\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:25.791329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.791192 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efed0ff1-n5m2v\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:25.800398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.799748 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efed0ff1-n5m2v\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:25.814821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.814797 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ksbk\" (UniqueName: \"kubernetes.io/projected/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-kube-api-access-6ksbk\") pod \"maas-api-revoke-sub-efed0ff1-n5m2v\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:25.912493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:25.912461 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:26.705400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:26.699412 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v"] Oct 30 06:11:27.055386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:27.054700 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" event={"ID":"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1","Type":"ContainerStarted","Data":"ec17090ade45b7440766e9e6ed416f51ef1b3ed523ff59e547cd0d038c259f82"} Oct 30 06:11:27.237331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:27.232460 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:27.663532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:27.663430 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:27.921260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:27.921178 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c"] Oct 30 06:11:27.925323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:27.925289 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:27.946380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:27.946351 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c"] Oct 30 06:11:28.013762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.013724 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qt8\" (UniqueName: \"kubernetes.io/projected/2d0514ee-2b01-4075-b812-c3680de46879-kube-api-access-b2qt8\") pod \"maas-api-revoke-sub-5fda812d-hzv7c\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:28.013762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.013769 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fda812d-hzv7c\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:28.013978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.013801 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fda812d-hzv7c\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:28.066686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.066646 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" event={"ID":"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1","Type":"ContainerStarted","Data":"8b1187ff20cde8189a2cb11693387532fdef1adb84380c172b288a6c7da6cddc"} Oct 30 06:11:28.109240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.109186 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" podStartSLOduration=3.109174121 podStartE2EDuration="3.109174121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:28.107683283 +0000 UTC m=+1102.103640294" watchObservedRunningTime="2026-10-30 06:11:28.109174121 +0000 UTC m=+1102.105131135" Oct 30 06:11:28.115677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.115647 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qt8\" (UniqueName: \"kubernetes.io/projected/2d0514ee-2b01-4075-b812-c3680de46879-kube-api-access-b2qt8\") pod \"maas-api-revoke-sub-5fda812d-hzv7c\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:28.115838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.115700 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fda812d-hzv7c\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:28.115838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.115813 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fda812d-hzv7c\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:28.116442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.116425 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fda812d-hzv7c\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:28.118709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.118692 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fda812d-hzv7c\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:28.139856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.139834 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qt8\" (UniqueName: \"kubernetes.io/projected/2d0514ee-2b01-4075-b812-c3680de46879-kube-api-access-b2qt8\") pod \"maas-api-revoke-sub-5fda812d-hzv7c\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:28.239180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.239149 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:28.544243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:28.544212 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c"] Oct 30 06:11:28.545163 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:28.545141 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0514ee_2b01_4075_b812_c3680de46879.slice/crio-aa7d94b5bf1ada2f7e3e34d6a2f566af5e49926de07b7e4c5ad4c41ec52cdbeb WatchSource:0}: Error finding container aa7d94b5bf1ada2f7e3e34d6a2f566af5e49926de07b7e4c5ad4c41ec52cdbeb: Status 404 returned error can't find the container with id aa7d94b5bf1ada2f7e3e34d6a2f566af5e49926de07b7e4c5ad4c41ec52cdbeb Oct 30 06:11:29.075840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:29.075801 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" event={"ID":"2d0514ee-2b01-4075-b812-c3680de46879","Type":"ContainerStarted","Data":"759242ef7088cbaf57fa9f2d346025d0aa2567831616b54bf9a1465a33b45d6f"} Oct 30 06:11:29.076363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:29.075868 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" event={"ID":"2d0514ee-2b01-4075-b812-c3680de46879","Type":"ContainerStarted","Data":"aa7d94b5bf1ada2f7e3e34d6a2f566af5e49926de07b7e4c5ad4c41ec52cdbeb"} Oct 30 06:11:29.077677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:29.077652 5001 generic.go:356] "Generic (PLEG): container finished" podID="c322a542-c1b4-4ba9-8ef6-58d062b9a7a1" containerID="8b1187ff20cde8189a2cb11693387532fdef1adb84380c172b288a6c7da6cddc" exitCode=0 Oct 30 06:11:29.077764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:29.077722 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" event={"ID":"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1","Type":"ContainerDied","Data":"8b1187ff20cde8189a2cb11693387532fdef1adb84380c172b288a6c7da6cddc"} Oct 30 06:11:29.120422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:29.120320 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" podStartSLOduration=2.120293987 podStartE2EDuration="2.120293987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:29.117011111 +0000 UTC m=+1103.112968123" watchObservedRunningTime="2026-10-30 06:11:29.120293987 +0000 UTC m=+1103.116250991" Oct 30 06:11:30.086494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.086461 5001 generic.go:356] "Generic (PLEG): container finished" podID="2d0514ee-2b01-4075-b812-c3680de46879" containerID="759242ef7088cbaf57fa9f2d346025d0aa2567831616b54bf9a1465a33b45d6f" exitCode=0 Oct 30 06:11:30.086945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.086561 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" event={"ID":"2d0514ee-2b01-4075-b812-c3680de46879","Type":"ContainerDied","Data":"759242ef7088cbaf57fa9f2d346025d0aa2567831616b54bf9a1465a33b45d6f"} Oct 30 06:11:30.260666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.260627 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:30.342022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.341935 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-xks-ca\") pod \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " Oct 30 06:11:30.342022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.342005 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-openshift-ca\") pod \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " Oct 30 06:11:30.342242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.342099 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-kube-api-access-6ksbk\" (UniqueName: \"kubernetes.io/projected/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-kube-api-access-6ksbk\") pod \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\" (UID: \"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1\") " Oct 30 06:11:30.342373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.342344 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-openshift-ca" pod "c322a542-c1b4-4ba9-8ef6-58d062b9a7a1" (UID: "c322a542-c1b4-4ba9-8ef6-58d062b9a7a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:30.342584 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.342569 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:30.344219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.344193 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-xks-ca" pod "c322a542-c1b4-4ba9-8ef6-58d062b9a7a1" (UID: "c322a542-c1b4-4ba9-8ef6-58d062b9a7a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:30.344375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.344262 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-kube-api-access-6ksbk" pod "c322a542-c1b4-4ba9-8ef6-58d062b9a7a1" (UID: "c322a542-c1b4-4ba9-8ef6-58d062b9a7a1"). InnerVolumeSpecName "kube-api-access-6ksbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:30.443653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.443620 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:30.443653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:30.443649 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ksbk\" (UniqueName: \"kubernetes.io/projected/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1-kube-api-access-6ksbk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:31.095621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.095583 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" Oct 30 06:11:31.095621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.095609 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v" event={"ID":"c322a542-c1b4-4ba9-8ef6-58d062b9a7a1","Type":"ContainerDied","Data":"ec17090ade45b7440766e9e6ed416f51ef1b3ed523ff59e547cd0d038c259f82"} Oct 30 06:11:31.096191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.095657 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec17090ade45b7440766e9e6ed416f51ef1b3ed523ff59e547cd0d038c259f82" Oct 30 06:11:31.254078 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.254048 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:31.353478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.353394 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0514ee-2b01-4075-b812-c3680de46879-kube-api-access-b2qt8\" (UniqueName: \"kubernetes.io/projected/2d0514ee-2b01-4075-b812-c3680de46879-kube-api-access-b2qt8\") pod \"2d0514ee-2b01-4075-b812-c3680de46879\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " Oct 30 06:11:31.353478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.353443 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-openshift-ca\") pod \"2d0514ee-2b01-4075-b812-c3680de46879\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " Oct 30 06:11:31.353699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.353521 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-xks-ca\") pod \"2d0514ee-2b01-4075-b812-c3680de46879\" (UID: \"2d0514ee-2b01-4075-b812-c3680de46879\") " Oct 30 06:11:31.354007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.353975 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-openshift-ca" pod "2d0514ee-2b01-4075-b812-c3680de46879" (UID: "2d0514ee-2b01-4075-b812-c3680de46879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:31.355942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.355913 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-xks-ca" pod "2d0514ee-2b01-4075-b812-c3680de46879" (UID: "2d0514ee-2b01-4075-b812-c3680de46879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:31.356071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.356018 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0514ee-2b01-4075-b812-c3680de46879-kube-api-access-b2qt8" pod "2d0514ee-2b01-4075-b812-c3680de46879" (UID: "2d0514ee-2b01-4075-b812-c3680de46879"). InnerVolumeSpecName "kube-api-access-b2qt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:31.477330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.458170 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:31.477646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.477622 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2qt8\" (UniqueName: \"kubernetes.io/projected/2d0514ee-2b01-4075-b812-c3680de46879-kube-api-access-b2qt8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:31.477785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:31.477769 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0514ee-2b01-4075-b812-c3680de46879-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:32.103189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:32.103159 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" Oct 30 06:11:32.103607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:32.103194 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c" event={"ID":"2d0514ee-2b01-4075-b812-c3680de46879","Type":"ContainerDied","Data":"aa7d94b5bf1ada2f7e3e34d6a2f566af5e49926de07b7e4c5ad4c41ec52cdbeb"} Oct 30 06:11:32.103607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:32.103243 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa7d94b5bf1ada2f7e3e34d6a2f566af5e49926de07b7e4c5ad4c41ec52cdbeb" Oct 30 06:11:34.948000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:34.947960 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz"] Oct 30 06:11:34.948524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:34.948513 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2d0514ee-2b01-4075-b812-c3680de46879" containerName="revoke-keys" Oct 30 06:11:34.948570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:34.948526 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0514ee-2b01-4075-b812-c3680de46879" containerName="revoke-keys" Oct 30 06:11:34.948570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:34.948535 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c322a542-c1b4-4ba9-8ef6-58d062b9a7a1" containerName="revoke-keys" Oct 30 06:11:34.948570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:34.948540 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c322a542-c1b4-4ba9-8ef6-58d062b9a7a1" containerName="revoke-keys" Oct 30 06:11:34.948666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:34.948640 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0514ee-2b01-4075-b812-c3680de46879" containerName="revoke-keys" Oct 30 06:11:34.948666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:34.948649 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c322a542-c1b4-4ba9-8ef6-58d062b9a7a1" containerName="revoke-keys" Oct 30 06:11:34.952433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:34.952416 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:34.979815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:34.979782 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz"] Oct 30 06:11:35.020002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.019964 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.020208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.020030 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.020208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.020125 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.020208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.020190 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.020401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.020292 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a443ba6-2255-490b-90b2-4922b8eaf75d-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.020514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.020480 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv626\" (UniqueName: \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-kube-api-access-fv626\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.020588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.020548 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.020697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.020671 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.020741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.020721 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.122172 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.122131 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.122372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.122175 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.122372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.122203 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.122372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.122269 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.122372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.122326 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.122604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.122372 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.122604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.122409 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a443ba6-2255-490b-90b2-4922b8eaf75d-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.122604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.122516 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv626\" (UniqueName: \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-kube-api-access-fv626\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.122604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.122559 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.122801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.122636 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.123014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.122985 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.123485 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.123353 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a443ba6-2255-490b-90b2-4922b8eaf75d-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.124011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.123777 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.124121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.124087 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.126508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.125934 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.126876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.126831 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.149192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.149166 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.149335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.149240 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv626\" (UniqueName: \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-kube-api-access-fv626\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.268875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.268792 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:35.598538 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:35.598505 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a443ba6_2255_490b_90b2_4922b8eaf75d.slice/crio-574b8b1ed2f9cd1374b373f87b6cea7134489faacfae8b1c5cfdc9a0e3a9d52a WatchSource:0}: Error finding container 574b8b1ed2f9cd1374b373f87b6cea7134489faacfae8b1c5cfdc9a0e3a9d52a: Status 404 returned error can't find the container with id 574b8b1ed2f9cd1374b373f87b6cea7134489faacfae8b1c5cfdc9a0e3a9d52a Oct 30 06:11:35.600059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.600026 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz"] Oct 30 06:11:35.601003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.600973 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:35.601088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.601034 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:35.601088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:35.601062 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:36.009994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.009958 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc"] Oct 30 06:11:36.014092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.014076 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:36.054924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.054891 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc"] Oct 30 06:11:36.055928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.055212 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:11:36.056106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.055296 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:11:36.056204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.055341 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:36.056289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.055398 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:11:36.137120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.137071 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" event={"ID":"1a443ba6-2255-490b-90b2-4922b8eaf75d","Type":"ContainerStarted","Data":"57f5e65e93e3fef5e23228f22ccb786a58bd46f2f5c3d58d71c5f26bfead1d93"} Oct 30 06:11:36.137350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.137133 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" event={"ID":"1a443ba6-2255-490b-90b2-4922b8eaf75d","Type":"ContainerStarted","Data":"574b8b1ed2f9cd1374b373f87b6cea7134489faacfae8b1c5cfdc9a0e3a9d52a"} Oct 30 06:11:36.137470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.137449 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c07a1f2-skfmc\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:36.137552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.137485 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c07a1f2-skfmc\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:36.138101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.137631 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7cpc\" (UniqueName: \"kubernetes.io/projected/34a0f834-6541-4317-9e9b-ffd9b0653e22-kube-api-access-l7cpc\") pod \"maas-api-revoke-sub-5c07a1f2-skfmc\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:36.185050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.184996 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" podStartSLOduration=2.184979997 podStartE2EDuration="2.184979997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:36.18391573 +0000 UTC m=+1110.179872722" watchObservedRunningTime="2026-10-30 06:11:36.184979997 +0000 UTC m=+1110.180936988" Oct 30 06:11:36.239400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.239362 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c07a1f2-skfmc\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:36.239611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.239415 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c07a1f2-skfmc\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:36.239802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.239780 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7cpc\" (UniqueName: \"kubernetes.io/projected/34a0f834-6541-4317-9e9b-ffd9b0653e22-kube-api-access-l7cpc\") pod \"maas-api-revoke-sub-5c07a1f2-skfmc\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:36.240170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.240153 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c07a1f2-skfmc\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:36.242421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.242395 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c07a1f2-skfmc\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:36.273387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.270694 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:36.280364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.274334 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:11:36.280364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.274407 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 06:11:36.280364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.279922 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7cpc\" (UniqueName: \"kubernetes.io/projected/34a0f834-6541-4317-9e9b-ffd9b0653e22-kube-api-access-l7cpc\") pod \"maas-api-revoke-sub-5c07a1f2-skfmc\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:36.328456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.328424 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:36.684990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:36.684962 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc"] Oct 30 06:11:37.146082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:37.146046 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" event={"ID":"34a0f834-6541-4317-9e9b-ffd9b0653e22","Type":"ContainerStarted","Data":"d0dbbccfb4e0a835d48eb66e5bda5c79632570dc7eedb19e60468e7991c4dc02"} Oct 30 06:11:37.146513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:37.146112 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" event={"ID":"34a0f834-6541-4317-9e9b-ffd9b0653e22","Type":"ContainerStarted","Data":"a80e26bd38841a83d01f6adc712d37b65ed2043a9debd4367ebeb7de5a466b1e"} Oct 30 06:11:37.199141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:37.198729 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" podStartSLOduration=2.19871332 podStartE2EDuration="2.19871332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:37.193192562 +0000 UTC m=+1111.189149572" watchObservedRunningTime="2026-10-30 06:11:37.19871332 +0000 UTC m=+1111.194670354" Oct 30 06:11:37.270377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:37.270263 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:11:37.270554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:37.270381 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 06:11:38.178640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.178605 5001 generic.go:356] "Generic (PLEG): container finished" podID="34a0f834-6541-4317-9e9b-ffd9b0653e22" containerID="d0dbbccfb4e0a835d48eb66e5bda5c79632570dc7eedb19e60468e7991c4dc02" exitCode=0 Oct 30 06:11:38.179661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.179636 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" event={"ID":"34a0f834-6541-4317-9e9b-ffd9b0653e22","Type":"ContainerDied","Data":"d0dbbccfb4e0a835d48eb66e5bda5c79632570dc7eedb19e60468e7991c4dc02"} Oct 30 06:11:38.277331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.276606 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:11:38.277331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.276668 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 06:11:38.293508 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:11:38.293468 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:11:38.793349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.792428 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr"] Oct 30 06:11:38.803852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.803822 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr"] Oct 30 06:11:38.804041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.803957 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:38.821365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.821343 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"distinct-2-w1-8d44dd-kserve-self-signed-certs\"" Oct 30 06:11:38.901018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.900984 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-model-cache\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:38.901238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.901223 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-dshm\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:38.901349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.901339 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-tmp-dir\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:38.901442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.901432 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a117db9-0054-4f46-a27d-c3861be4ca0e-tls-certs\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:38.901672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.901657 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-home\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:38.901807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:38.901787 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzp7m\" (UniqueName: \"kubernetes.io/projected/7a117db9-0054-4f46-a27d-c3861be4ca0e-kube-api-access-qzp7m\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.004866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.004015 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-home\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.004866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.004074 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzp7m\" (UniqueName: \"kubernetes.io/projected/7a117db9-0054-4f46-a27d-c3861be4ca0e-kube-api-access-qzp7m\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.004866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.004180 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-model-cache\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.004866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.004210 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-dshm\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.004866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.004258 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-tmp-dir\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.004866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.004323 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a117db9-0054-4f46-a27d-c3861be4ca0e-tls-certs\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.005705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.005683 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-model-cache\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.006340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.006268 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-tmp-dir\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.007508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.007253 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-home\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.016864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.016840 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-dshm\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.044388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.043355 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a117db9-0054-4f46-a27d-c3861be4ca0e-tls-certs\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.053151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.053107 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzp7m\" (UniqueName: \"kubernetes.io/projected/7a117db9-0054-4f46-a27d-c3861be4ca0e-kube-api-access-qzp7m\") pod \"distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.119899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.119869 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:39.289333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.274693 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:11:39.289333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.274763 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 06:11:39.495464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.495433 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr"] Oct 30 06:11:39.497779 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:39.497737 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a117db9_0054_4f46_a27d_c3861be4ca0e.slice/crio-fe511eb97f82bd2b9f8e15b0e73645c789a794fb9b56ab0057d6ce315896eaf9 WatchSource:0}: Error finding container fe511eb97f82bd2b9f8e15b0e73645c789a794fb9b56ab0057d6ce315896eaf9: Status 404 returned error can't find the container with id fe511eb97f82bd2b9f8e15b0e73645c789a794fb9b56ab0057d6ce315896eaf9 Oct 30 06:11:39.511623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.511591 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:39.613994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.613961 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-openshift-ca\") pod \"34a0f834-6541-4317-9e9b-ffd9b0653e22\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " Oct 30 06:11:39.614181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.614054 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a0f834-6541-4317-9e9b-ffd9b0653e22-kube-api-access-l7cpc\" (UniqueName: \"kubernetes.io/projected/34a0f834-6541-4317-9e9b-ffd9b0653e22-kube-api-access-l7cpc\") pod \"34a0f834-6541-4317-9e9b-ffd9b0653e22\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " Oct 30 06:11:39.614457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.614426 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-openshift-ca" pod "34a0f834-6541-4317-9e9b-ffd9b0653e22" (UID: "34a0f834-6541-4317-9e9b-ffd9b0653e22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:39.614670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.614651 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-xks-ca\") pod \"34a0f834-6541-4317-9e9b-ffd9b0653e22\" (UID: \"34a0f834-6541-4317-9e9b-ffd9b0653e22\") " Oct 30 06:11:39.615319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.615236 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:39.616355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.616332 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a0f834-6541-4317-9e9b-ffd9b0653e22-kube-api-access-l7cpc" pod "34a0f834-6541-4317-9e9b-ffd9b0653e22" (UID: "34a0f834-6541-4317-9e9b-ffd9b0653e22"). InnerVolumeSpecName "kube-api-access-l7cpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:39.616816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.616796 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-xks-ca" pod "34a0f834-6541-4317-9e9b-ffd9b0653e22" (UID: "34a0f834-6541-4317-9e9b-ffd9b0653e22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:39.716568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.716532 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7cpc\" (UniqueName: \"kubernetes.io/projected/34a0f834-6541-4317-9e9b-ffd9b0653e22-kube-api-access-l7cpc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:39.716568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:39.716559 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a0f834-6541-4317-9e9b-ffd9b0653e22-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:40.207588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.207544 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" event={"ID":"7a117db9-0054-4f46-a27d-c3861be4ca0e","Type":"ContainerStarted","Data":"9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da"} Oct 30 06:11:40.207767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.207603 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" event={"ID":"7a117db9-0054-4f46-a27d-c3861be4ca0e","Type":"ContainerStarted","Data":"fe511eb97f82bd2b9f8e15b0e73645c789a794fb9b56ab0057d6ce315896eaf9"} Oct 30 06:11:40.208975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.208948 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" Oct 30 06:11:40.209059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.208986 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc" event={"ID":"34a0f834-6541-4317-9e9b-ffd9b0653e22","Type":"ContainerDied","Data":"a80e26bd38841a83d01f6adc712d37b65ed2043a9debd4367ebeb7de5a466b1e"} Oct 30 06:11:40.209059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.209041 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a80e26bd38841a83d01f6adc712d37b65ed2043a9debd4367ebeb7de5a466b1e" Oct 30 06:11:40.251102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.251054 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podStartSLOduration=2.251038957 podStartE2EDuration="2.251038957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:40.249388157 +0000 UTC m=+1114.245345180" watchObservedRunningTime="2026-10-30 06:11:40.251038957 +0000 UTC m=+1114.246995966" Oct 30 06:11:40.270122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.269680 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:11:40.270122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.269734 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 06:11:40.498179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.498126 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb"] Oct 30 06:11:40.498984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.498960 5001 cpu_manager.go:414] "containerMap: removing container" podUID="34a0f834-6541-4317-9e9b-ffd9b0653e22" containerName="revoke-keys" Oct 30 06:11:40.499107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.498986 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a0f834-6541-4317-9e9b-ffd9b0653e22" containerName="revoke-keys" Oct 30 06:11:40.499181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.499159 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a0f834-6541-4317-9e9b-ffd9b0653e22" containerName="revoke-keys" Oct 30 06:11:40.503337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.503295 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:40.520163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.520100 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb"] Oct 30 06:11:40.521605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.521579 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:40.521724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.521623 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:11:40.521878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.521857 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:11:40.521984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.521921 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:11:40.630779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.630750 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d630e74-h5pxb\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:40.630930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.630839 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxkbn\" (UniqueName: \"kubernetes.io/projected/213d154b-abcb-4d71-9def-bc2b2aa6570f-kube-api-access-sxkbn\") pod \"maas-api-revoke-sub-9d630e74-h5pxb\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:40.630930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.630886 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d630e74-h5pxb\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:40.633226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.633204 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf"] Oct 30 06:11:40.637625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.637609 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.671341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.664688 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a519e9-openshift-default-dockercfg-nffns\"" Oct 30 06:11:40.699276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.699244 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf"] Oct 30 06:11:40.732837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.732803 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.732988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.732862 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxkbn\" (UniqueName: \"kubernetes.io/projected/213d154b-abcb-4d71-9def-bc2b2aa6570f-kube-api-access-sxkbn\") pod \"maas-api-revoke-sub-9d630e74-h5pxb\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:40.732988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.732939 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d630e74-h5pxb\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:40.733069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.733004 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.733069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.733030 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.733138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.733094 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd447616-cb5a-417a-994f-25acea127f8f-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.733138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.733129 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.733197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.733177 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8znsm\" (UniqueName: \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-kube-api-access-8znsm\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.733232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.733208 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.733270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.733241 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd447616-cb5a-417a-994f-25acea127f8f-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.733319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.733291 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d630e74-h5pxb\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:40.733375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.733355 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.734414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.734391 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d630e74-h5pxb\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:40.739604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.739577 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d630e74-h5pxb\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:40.789475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.789400 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxkbn\" (UniqueName: \"kubernetes.io/projected/213d154b-abcb-4d71-9def-bc2b2aa6570f-kube-api-access-sxkbn\") pod \"maas-api-revoke-sub-9d630e74-h5pxb\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:40.819082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.819057 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:40.834603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.834578 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.834701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.834653 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.834780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.834767 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.834824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.834795 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.834869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.834855 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd447616-cb5a-417a-994f-25acea127f8f-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.834907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.834895 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.836008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.834944 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8znsm\" (UniqueName: \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-kube-api-access-8znsm\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.836008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.834976 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.836008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.835006 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd447616-cb5a-417a-994f-25acea127f8f-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.836008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.835225 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.836008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.835769 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.836008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.835943 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.836243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.836101 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd447616-cb5a-417a-994f-25acea127f8f-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.836418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.836398 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.837858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.837388 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.838348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.838269 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd447616-cb5a-417a-994f-25acea127f8f-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.889149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.889113 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8znsm\" (UniqueName: \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-kube-api-access-8znsm\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.889737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.889718 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:40.967132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:40.966536 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:41.120186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:41.120140 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:41.194909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:41.194885 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb"] Oct 30 06:11:41.200729 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:41.200683 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213d154b_abcb_4d71_9def_bc2b2aa6570f.slice/crio-4eea882cad24b17e86b20244d7c571564c82171634bfc9690dd90b456f303604 WatchSource:0}: Error finding container 4eea882cad24b17e86b20244d7c571564c82171634bfc9690dd90b456f303604: Status 404 returned error can't find the container with id 4eea882cad24b17e86b20244d7c571564c82171634bfc9690dd90b456f303604 Oct 30 06:11:41.218404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:41.218373 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" event={"ID":"213d154b-abcb-4d71-9def-bc2b2aa6570f","Type":"ContainerStarted","Data":"4eea882cad24b17e86b20244d7c571564c82171634bfc9690dd90b456f303604"} Oct 30 06:11:41.282334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:41.280738 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:11:41.282334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:41.280795 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 06:11:41.388643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:41.388614 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf"] Oct 30 06:11:41.397108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:41.397062 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:41.397333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:41.397285 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:41.397518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:41.397496 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:42.227794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:42.227757 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" event={"ID":"fd447616-cb5a-417a-994f-25acea127f8f","Type":"ContainerStarted","Data":"65347044dfc1dcf85109df612336f773f80e88e56c168b77da2f49ca52d7f754"} Oct 30 06:11:42.228275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:42.227827 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" event={"ID":"fd447616-cb5a-417a-994f-25acea127f8f","Type":"ContainerStarted","Data":"b2bd1a338f555e17b634e57f6dc0bfdfd054a8cf329a861492feeb64ece47edf"} Oct 30 06:11:42.229587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:42.229557 5001 generic.go:356] "Generic (PLEG): container finished" podID="213d154b-abcb-4d71-9def-bc2b2aa6570f" containerID="8a964246a75863c36a41ba02e7a4524b17fa4cd727d70247f7949a2be11c0914" exitCode=0 Oct 30 06:11:42.229694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:42.229629 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" event={"ID":"213d154b-abcb-4d71-9def-bc2b2aa6570f","Type":"ContainerDied","Data":"8a964246a75863c36a41ba02e7a4524b17fa4cd727d70247f7949a2be11c0914"} Oct 30 06:11:42.294348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:42.291469 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:11:42.294348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:42.291531 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 06:11:42.309343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:42.309275 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" podStartSLOduration=2.309260483 podStartE2EDuration="2.309260483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:42.302507893 +0000 UTC m=+1116.298464906" watchObservedRunningTime="2026-10-30 06:11:42.309260483 +0000 UTC m=+1116.305217493" Oct 30 06:11:42.972546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:42.972223 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:42.972546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:42.972383 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 06:11:42.972546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:42.972429 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" podUID="fd447616-cb5a-417a-994f-25acea127f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 06:11:43.287367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.287171 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:43.288403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.288297 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:43.289289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.289266 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:11:43.505067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.504997 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:43.549908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.549871 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:11:43.604933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.603103 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-xks-ca\") pod \"213d154b-abcb-4d71-9def-bc2b2aa6570f\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " Oct 30 06:11:43.604933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.603169 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-openshift-ca\") pod \"213d154b-abcb-4d71-9def-bc2b2aa6570f\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " Oct 30 06:11:43.604933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.603221 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213d154b-abcb-4d71-9def-bc2b2aa6570f-kube-api-access-sxkbn\" (UniqueName: \"kubernetes.io/projected/213d154b-abcb-4d71-9def-bc2b2aa6570f-kube-api-access-sxkbn\") pod \"213d154b-abcb-4d71-9def-bc2b2aa6570f\" (UID: \"213d154b-abcb-4d71-9def-bc2b2aa6570f\") " Oct 30 06:11:43.604933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.604501 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-openshift-ca" pod "213d154b-abcb-4d71-9def-bc2b2aa6570f" (UID: "213d154b-abcb-4d71-9def-bc2b2aa6570f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:43.606680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.606654 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213d154b-abcb-4d71-9def-bc2b2aa6570f-kube-api-access-sxkbn" pod "213d154b-abcb-4d71-9def-bc2b2aa6570f" (UID: "213d154b-abcb-4d71-9def-bc2b2aa6570f"). InnerVolumeSpecName "kube-api-access-sxkbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:43.606916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.606897 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-xks-ca" pod "213d154b-abcb-4d71-9def-bc2b2aa6570f" (UID: "213d154b-abcb-4d71-9def-bc2b2aa6570f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:43.699324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.698768 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6"] Oct 30 06:11:43.734330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.703139 5001 cpu_manager.go:414] "containerMap: removing container" podUID="213d154b-abcb-4d71-9def-bc2b2aa6570f" containerName="revoke-keys" Oct 30 06:11:43.734330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.722685 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213d154b-abcb-4d71-9def-bc2b2aa6570f" containerName="revoke-keys" Oct 30 06:11:43.734330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.722937 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="213d154b-abcb-4d71-9def-bc2b2aa6570f" containerName="revoke-keys" Oct 30 06:11:43.747332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.705711 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:43.747332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.735499 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213d154b-abcb-4d71-9def-bc2b2aa6570f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:43.747332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.735519 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxkbn\" (UniqueName: \"kubernetes.io/projected/213d154b-abcb-4d71-9def-bc2b2aa6570f-kube-api-access-sxkbn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:43.807380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.807239 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6"] Oct 30 06:11:43.807551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.807407 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:43.836611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.836526 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da240f3f-bpbj6\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:43.836752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.836671 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da240f3f-bpbj6\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:43.836906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.836878 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9lqq\" (UniqueName: \"kubernetes.io/projected/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-kube-api-access-b9lqq\") pod \"maas-api-revoke-sub-da240f3f-bpbj6\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:43.937817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.937780 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9lqq\" (UniqueName: \"kubernetes.io/projected/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-kube-api-access-b9lqq\") pod \"maas-api-revoke-sub-da240f3f-bpbj6\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:43.937990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.937944 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da240f3f-bpbj6\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:43.937990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.937972 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da240f3f-bpbj6\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:43.938622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.938601 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da240f3f-bpbj6\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:43.940487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.940470 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da240f3f-bpbj6\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:43.971330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.970256 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 06:11:43.971330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.970356 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" podUID="fd447616-cb5a-417a-994f-25acea127f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 06:11:43.982334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:43.980965 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9lqq\" (UniqueName: \"kubernetes.io/projected/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-kube-api-access-b9lqq\") pod \"maas-api-revoke-sub-da240f3f-bpbj6\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:44.151349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:44.149279 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:44.324336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:44.315423 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" event={"ID":"213d154b-abcb-4d71-9def-bc2b2aa6570f","Type":"ContainerDied","Data":"4eea882cad24b17e86b20244d7c571564c82171634bfc9690dd90b456f303604"} Oct 30 06:11:44.324897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:44.324869 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eea882cad24b17e86b20244d7c571564c82171634bfc9690dd90b456f303604" Oct 30 06:11:44.325023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:44.319357 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb" Oct 30 06:11:45.066339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:45.061552 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 06:11:45.068994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:45.068958 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" podUID="fd447616-cb5a-417a-994f-25acea127f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 06:11:45.077597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:45.077564 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6"] Oct 30 06:11:45.361165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:45.361077 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" event={"ID":"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5","Type":"ContainerStarted","Data":"a384023cbfa58756f0cfa495f6ca5665e57890b943609d75730a535f34a9c33c"} Oct 30 06:11:45.968333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:45.968057 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 06:11:45.968333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:45.968133 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" podUID="fd447616-cb5a-417a-994f-25acea127f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 06:11:46.117333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.115429 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:46.148332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.147770 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:46.162447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.162404 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:46.287525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.287454 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:46.366689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.365418 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 06:11:46.421398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.420343 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" event={"ID":"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5","Type":"ContainerStarted","Data":"c58d049958aaa3532df5b8aa5a1a34547c31b94355383c22f4360373ce2071a3"} Oct 30 06:11:46.526334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.521332 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" podStartSLOduration=3.521293139 podStartE2EDuration="3.521293139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:46.494675258 +0000 UTC m=+1120.490632270" watchObservedRunningTime="2026-10-30 06:11:46.521293139 +0000 UTC m=+1120.517250150" Oct 30 06:11:46.548744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.548676 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jzt6v-mwmx8"] Oct 30 06:11:46.579223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.579186 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.687449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.642262 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:11:46.788329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.787381 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:46.788329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.652142 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jzt6v-mwmx8"] Oct 30 06:11:46.788329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.756524 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-service-ca-bundle\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.788329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.787621 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t287\" (UniqueName: \"kubernetes.io/projected/ecabf567-9ebd-4831-ad26-6ad20065fdbf-kube-api-access-8t287\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.788329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.787796 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ecabf567-9ebd-4831-ad26-6ad20065fdbf-archives-path\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.788329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.787839 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-trusted-ca-bundle\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.788329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.648687 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dk4qd\"" Oct 30 06:11:46.908367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.908139 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ecabf567-9ebd-4831-ad26-6ad20065fdbf-archives-path\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.908622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.908601 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-trusted-ca-bundle\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.908807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.908793 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-service-ca-bundle\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.908992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.908978 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t287\" (UniqueName: \"kubernetes.io/projected/ecabf567-9ebd-4831-ad26-6ad20065fdbf-kube-api-access-8t287\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.910138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.910115 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ecabf567-9ebd-4831-ad26-6ad20065fdbf-archives-path\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.911468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.911447 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-trusted-ca-bundle\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.912210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.912159 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-service-ca-bundle\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.967670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.967581 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 06:11:46.967963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.967933 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" podUID="fd447616-cb5a-417a-994f-25acea127f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 06:11:46.994336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.993951 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t287\" (UniqueName: \"kubernetes.io/projected/ecabf567-9ebd-4831-ad26-6ad20065fdbf-kube-api-access-8t287\") pod \"periodic-gathering-jzt6v-mwmx8\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:46.996960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:46.996744 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:11:47.631331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:47.623753 5001 generic.go:356] "Generic (PLEG): container finished" podID="abf4e00d-0125-46b9-b0d3-1a98ca43d1c5" containerID="c58d049958aaa3532df5b8aa5a1a34547c31b94355383c22f4360373ce2071a3" exitCode=0 Oct 30 06:11:47.631331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:47.624036 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" event={"ID":"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5","Type":"ContainerDied","Data":"c58d049958aaa3532df5b8aa5a1a34547c31b94355383c22f4360373ce2071a3"} Oct 30 06:11:47.819428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:47.819206 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jzt6v-mwmx8"] Oct 30 06:11:47.981291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:47.981257 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 06:11:47.981582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:47.981554 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" podUID="fd447616-cb5a-417a-994f-25acea127f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 06:11:48.633772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:48.633729 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" event={"ID":"ecabf567-9ebd-4831-ad26-6ad20065fdbf","Type":"ContainerStarted","Data":"42832c55ea98d61eae218eda97c70578a13370bad70f89d627e01f5e21b701f0"} Oct 30 06:11:48.634210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:48.633797 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" event={"ID":"ecabf567-9ebd-4831-ad26-6ad20065fdbf","Type":"ContainerStarted","Data":"4eb29e0d15ea07f2726bf92f49097cc996da3ff00033fd3d314b3422bb1e6ffd"} Oct 30 06:11:48.966542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:48.966512 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:48.967298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:48.967268 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 06:11:48.967510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:48.967482 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" podUID="fd447616-cb5a-417a-994f-25acea127f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 06:11:49.017206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.017170 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-xks-ca\") pod \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " Oct 30 06:11:49.017432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.017356 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" podStartSLOduration=3.017339785 podStartE2EDuration="3.017339785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:48.787829609 +0000 UTC m=+1122.783786620" watchObservedRunningTime="2026-10-30 06:11:49.017339785 +0000 UTC m=+1123.013296795" Oct 30 06:11:49.017432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.017394 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-openshift-ca\") pod \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " Oct 30 06:11:49.017432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.017421 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-kube-api-access-b9lqq\" (UniqueName: \"kubernetes.io/projected/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-kube-api-access-b9lqq\") pod \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\" (UID: \"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5\") " Oct 30 06:11:49.018402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.018371 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-openshift-ca" pod "abf4e00d-0125-46b9-b0d3-1a98ca43d1c5" (UID: "abf4e00d-0125-46b9-b0d3-1a98ca43d1c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:49.020806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.020771 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-kube-api-access-b9lqq" pod "abf4e00d-0125-46b9-b0d3-1a98ca43d1c5" (UID: "abf4e00d-0125-46b9-b0d3-1a98ca43d1c5"). InnerVolumeSpecName "kube-api-access-b9lqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:49.022424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.022401 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-xks-ca" pod "abf4e00d-0125-46b9-b0d3-1a98ca43d1c5" (UID: "abf4e00d-0125-46b9-b0d3-1a98ca43d1c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:49.076388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.076349 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br"] Oct 30 06:11:49.077186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.077165 5001 cpu_manager.go:414] "containerMap: removing container" podUID="abf4e00d-0125-46b9-b0d3-1a98ca43d1c5" containerName="revoke-keys" Oct 30 06:11:49.077274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.077189 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf4e00d-0125-46b9-b0d3-1a98ca43d1c5" containerName="revoke-keys" Oct 30 06:11:49.077438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.077422 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf4e00d-0125-46b9-b0d3-1a98ca43d1c5" containerName="revoke-keys" Oct 30 06:11:49.081716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.081691 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.105481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.105452 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br"] Oct 30 06:11:49.105698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.105679 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ba44b248-openshift-default-dockercfg-2z2v7\"" Oct 30 06:11:49.118332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118273 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-envoy\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.118460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118333 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-podinfo\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.118460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118441 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-data\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.118579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118540 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j796k\" (UniqueName: \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-kube-api-access-j796k\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.118633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118577 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-socket\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.118633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118614 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istiod-ca-cert\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.118740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118684 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-certs\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.118740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118729 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-credential-socket\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.118844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118767 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-token\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.118900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118847 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:49.118900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118865 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9lqq\" (UniqueName: \"kubernetes.io/projected/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-kube-api-access-b9lqq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:49.118900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.118880 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:11:49.120430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.120410 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:49.141325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.140730 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:49.227507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.227095 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j796k\" (UniqueName: \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-kube-api-access-j796k\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.227507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.227154 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-socket\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.227507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.227200 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istiod-ca-cert\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.227507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.227275 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-certs\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.227507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.227343 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-credential-socket\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.227507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.227382 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-token\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.227507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.227465 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-envoy\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.227507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.227490 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-podinfo\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.228083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.227521 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-data\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.228083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.228052 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-data\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.229124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.228627 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-socket\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.251337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.248747 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-credential-socket\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.251337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.249215 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istiod-ca-cert\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.267336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.256805 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-certs\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.289949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.280388 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j796k\" (UniqueName: \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-kube-api-access-j796k\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.289949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.282696 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-envoy\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.289949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.288759 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-token\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.303333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.300370 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-podinfo\") pod \"e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.409362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.406872 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:49.547175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.547141 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd"] Oct 30 06:11:49.552891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.552857 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.583217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.583177 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd"] Oct 30 06:11:49.644407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.644358 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" event={"ID":"abf4e00d-0125-46b9-b0d3-1a98ca43d1c5","Type":"ContainerDied","Data":"a384023cbfa58756f0cfa495f6ca5665e57890b943609d75730a535f34a9c33c"} Oct 30 06:11:49.644921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.644456 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a384023cbfa58756f0cfa495f6ca5665e57890b943609d75730a535f34a9c33c" Oct 30 06:11:49.644921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.644377 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6" Oct 30 06:11:49.668261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.664690 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:11:49.738893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.738797 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.739065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.739035 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.739146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.739084 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.739213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.739148 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d25535-ffbc-4239-abb4-9dca02a1b62e-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.739270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.739236 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.739376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.739271 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlh7d\" (UniqueName: \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-kube-api-access-nlh7d\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.739376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.739295 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.739510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.739438 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.739510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.739459 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.757060 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:49.757027 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ac2989_2bfd_4b85_9fbf_c3c04f3b2c39.slice/crio-118fc22151ac3a3548e13ce23ffc8a0205c4f4225bc73130cad0009b14cd0430 WatchSource:0}: Error finding container 118fc22151ac3a3548e13ce23ffc8a0205c4f4225bc73130cad0009b14cd0430: Status 404 returned error can't find the container with id 118fc22151ac3a3548e13ce23ffc8a0205c4f4225bc73130cad0009b14cd0430 Oct 30 06:11:49.759526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.759484 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:49.759642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.759576 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:49.759642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.759615 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:49.765599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.763832 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br"] Oct 30 06:11:49.840383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.840346 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.840503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.840424 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlh7d\" (UniqueName: \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-kube-api-access-nlh7d\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.840503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.840463 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.840604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.840553 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.840604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.840582 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.840719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.840631 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.840775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.840723 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.840829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.840792 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.840878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.840830 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.840878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.840836 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.840878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.840859 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d25535-ffbc-4239-abb4-9dca02a1b62e-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.841494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.841044 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.841494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.841088 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.841494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.841435 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d25535-ffbc-4239-abb4-9dca02a1b62e-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.843535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.843511 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.844406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.844385 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.866220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.866136 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.867174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.867150 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlh7d\" (UniqueName: \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-kube-api-access-nlh7d\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:49.868610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:49.868294 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:50.016446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.016361 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz"] Oct 30 06:11:50.023415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.023387 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.044965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.044913 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz"] Oct 30 06:11:50.149278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.149242 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.149488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.149342 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.149488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.149366 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9wzg\" (UniqueName: \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-kube-api-access-f9wzg\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.149706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.149520 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.149706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.149562 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.149706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.149593 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.149706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.149678 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.149865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.149775 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.149865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.149805 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.238236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.238197 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd"] Oct 30 06:11:50.245391 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:50.245366 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d25535_ffbc_4239_abb4_9dca02a1b62e.slice/crio-67e856dd63ec842f924480adb5a5d02c20cdcd3db4992e4b2138e16cf8cca853 WatchSource:0}: Error finding container 67e856dd63ec842f924480adb5a5d02c20cdcd3db4992e4b2138e16cf8cca853: Status 404 returned error can't find the container with id 67e856dd63ec842f924480adb5a5d02c20cdcd3db4992e4b2138e16cf8cca853 Oct 30 06:11:50.248837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.248799 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:50.248911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.248885 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:50.248957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.248927 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:50.256766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.256735 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.256957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.256785 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9wzg\" (UniqueName: \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-kube-api-access-f9wzg\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.256957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.256834 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.256957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.256868 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.256957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.256896 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.256957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.256947 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.258965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.257067 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.258965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.257112 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.258965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.257877 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.259171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.259148 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.259716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.259696 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.259855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.259723 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.260006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.259899 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.260650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.260614 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.262641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.262619 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.264965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.264942 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.281815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.281788 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.283583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.283556 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9wzg\" (UniqueName: \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-kube-api-access-f9wzg\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.355211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.355032 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:50.386454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.386426 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:50.387764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.387739 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:50.389414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.388952 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:11:50.662088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.661243 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" event={"ID":"81d25535-ffbc-4239-abb4-9dca02a1b62e","Type":"ContainerStarted","Data":"285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe"} Oct 30 06:11:50.662088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.661331 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" event={"ID":"81d25535-ffbc-4239-abb4-9dca02a1b62e","Type":"ContainerStarted","Data":"67e856dd63ec842f924480adb5a5d02c20cdcd3db4992e4b2138e16cf8cca853"} Oct 30 06:11:50.664386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.664353 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" event={"ID":"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39","Type":"ContainerStarted","Data":"042d9c4c3e3598227335bc182d22baa9e8e698eef9278c5f3f985434feff7711"} Oct 30 06:11:50.664567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.664409 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" event={"ID":"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39","Type":"ContainerStarted","Data":"118fc22151ac3a3548e13ce23ffc8a0205c4f4225bc73130cad0009b14cd0430"} Oct 30 06:11:50.705278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.705252 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz"] Oct 30 06:11:50.705653 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:50.705622 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cec6bc_bd19_4c3b_9520_8fd3eb41c005.slice/crio-da206877c07f9a6a5cd79ce533fbd5d0cf62a72d5f9d5ae8e40066b742cfba0e WatchSource:0}: Error finding container da206877c07f9a6a5cd79ce533fbd5d0cf62a72d5f9d5ae8e40066b742cfba0e: Status 404 returned error can't find the container with id da206877c07f9a6a5cd79ce533fbd5d0cf62a72d5f9d5ae8e40066b742cfba0e Oct 30 06:11:50.708623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.708584 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" podStartSLOduration=1.7085703209999998 podStartE2EDuration="1.708570321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:50.707719451 +0000 UTC m=+1124.703676461" watchObservedRunningTime="2026-10-30 06:11:50.708570321 +0000 UTC m=+1124.704527330" Oct 30 06:11:50.708812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.708772 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:50.708889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.708863 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:50.708942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.708909 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:50.756842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.756785 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" podStartSLOduration=1.756764292 podStartE2EDuration="1.756764292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:50.754608831 +0000 UTC m=+1124.750565852" watchObservedRunningTime="2026-10-30 06:11:50.756764292 +0000 UTC m=+1124.752721317" Oct 30 06:11:50.891746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:50.889402 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:51.409460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:51.409429 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:51.415708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:51.415681 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:51.719873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:51.719784 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" event={"ID":"43cec6bc-bd19-4c3b-9520-8fd3eb41c005","Type":"ContainerStarted","Data":"f97bcc009475f5b6960ac91e1838404d5943c34f480948f009226207e1050c60"} Oct 30 06:11:51.719873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:51.719845 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" event={"ID":"43cec6bc-bd19-4c3b-9520-8fd3eb41c005","Type":"ContainerStarted","Data":"da206877c07f9a6a5cd79ce533fbd5d0cf62a72d5f9d5ae8e40066b742cfba0e"} Oct 30 06:11:51.720632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:51.720611 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:51.721714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:51.721696 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:11:51.785330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:51.785117 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" podStartSLOduration=2.785094142 podStartE2EDuration="2.785094142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:51.770842753 +0000 UTC m=+1125.766799767" watchObservedRunningTime="2026-10-30 06:11:51.785094142 +0000 UTC m=+1125.781051153" Oct 30 06:11:51.874849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:51.874807 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 06:11:51.875134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:51.875102 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 06:11:52.368815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.368782 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:52.374433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.374404 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:11:52.374653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.374621 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 06:11:52.804509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.804400 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn"] Oct 30 06:11:52.813810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.813781 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:52.816151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.816122 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn"] Oct 30 06:11:52.894328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.894004 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 06:11:52.894328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.894157 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 06:11:52.974331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.929247 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:52.974614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.974582 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcadd15-1201-491a-8348-dd39b94f9bae-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:52.974823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.974805 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:52.975003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.974986 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:52.975125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.975110 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-697x6\" (UniqueName: \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-kube-api-access-697x6\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:52.975296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.975279 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:52.975466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.975445 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:52.975830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.975809 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:52.975988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:52.975968 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.084382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.081329 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.084382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.081403 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.084382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.081673 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.084382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.082199 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.084382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.082511 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.084382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.082588 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcadd15-1201-491a-8348-dd39b94f9bae-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.084382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.083457 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcadd15-1201-491a-8348-dd39b94f9bae-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.084382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.083572 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.084382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.083939 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.088659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.083616 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.089228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.088898 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-697x6\" (UniqueName: \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-kube-api-access-697x6\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.089469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.089452 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.107605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.107451 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.107800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.107775 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.115638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.115606 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.125173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.124799 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.128886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.128850 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.129084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.129000 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-697x6\" (UniqueName: \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-kube-api-access-697x6\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.134528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.134475 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:53.416457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.398641 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:11:53.416927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.416888 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 06:11:53.610792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.610763 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn"] Oct 30 06:11:53.613830 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:53.613791 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcadd15_1201_491a_8348_dd39b94f9bae.slice/crio-41691aaf33d95975bd1096f6937a8dec355936ddc8c49ae298e59dcd9cd331ac WatchSource:0}: Error finding container 41691aaf33d95975bd1096f6937a8dec355936ddc8c49ae298e59dcd9cd331ac: Status 404 returned error can't find the container with id 41691aaf33d95975bd1096f6937a8dec355936ddc8c49ae298e59dcd9cd331ac Oct 30 06:11:53.617137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.617081 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:53.617414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.617372 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:53.617608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.617580 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:53.736984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.736951 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" event={"ID":"1bcadd15-1201-491a-8348-dd39b94f9bae","Type":"ContainerStarted","Data":"41691aaf33d95975bd1096f6937a8dec355936ddc8c49ae298e59dcd9cd331ac"} Oct 30 06:11:53.871747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.871705 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 06:11:53.872355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:53.872289 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 06:11:54.356085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:54.356052 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:11:54.356377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:54.356349 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 06:11:54.838321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:54.838201 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" event={"ID":"1bcadd15-1201-491a-8348-dd39b94f9bae","Type":"ContainerStarted","Data":"15ce3dc383cb1eb254b2aed7d792942023c02235cfb13dff6491c01b1abfeeac"} Oct 30 06:11:54.874093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:54.874053 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 06:11:54.874732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:54.874699 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 06:11:54.943717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:54.943645 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" podStartSLOduration=2.9436236669999998 podStartE2EDuration="2.943623667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:54.940113256 +0000 UTC m=+1128.936070267" watchObservedRunningTime="2026-10-30 06:11:54.943623667 +0000 UTC m=+1128.939580678" Oct 30 06:11:55.142420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.142373 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:11:55.143074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.143043 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 06:11:55.143178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.143114 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" podUID="1bcadd15-1201-491a-8348-dd39b94f9bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 06:11:55.427998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.427900 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:11:55.428227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.428200 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 06:11:55.440694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.440658 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9"] Oct 30 06:11:55.575531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.575498 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9"] Oct 30 06:11:55.575838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.575821 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:11:55.619979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.619948 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ba44b248\"" Oct 30 06:11:55.651335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.639057 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ba44b248-dockercfg-llpg4\"" Oct 30 06:11:55.673328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.644260 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh"] Oct 30 06:11:55.685758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.684549 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc51944b-1255-4681-a6dc-fc6e503116b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9\" (UID: \"dc51944b-1255-4681-a6dc-fc6e503116b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:11:55.685758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.684730 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjbrn\" (UniqueName: \"kubernetes.io/projected/dc51944b-1255-4681-a6dc-fc6e503116b5-kube-api-access-hjbrn\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9\" (UID: \"dc51944b-1255-4681-a6dc-fc6e503116b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:11:55.813332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.793523 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh"] Oct 30 06:11:55.813332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.793663 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:11:55.813332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.808191 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc51944b-1255-4681-a6dc-fc6e503116b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9\" (UID: \"dc51944b-1255-4681-a6dc-fc6e503116b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:11:55.813332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.808399 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjbrn\" (UniqueName: \"kubernetes.io/projected/dc51944b-1255-4681-a6dc-fc6e503116b5-kube-api-access-hjbrn\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9\" (UID: \"dc51944b-1255-4681-a6dc-fc6e503116b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:11:55.813332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.811604 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc51944b-1255-4681-a6dc-fc6e503116b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9\" (UID: \"dc51944b-1255-4681-a6dc-fc6e503116b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:11:55.921345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.917275 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-plugins-config-volume\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-kfvjh\" (UID: \"142a47dc-0a88-4e03-b0b7-8a62d8b8e861\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:11:55.922152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.922114 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s6tm\" (UniqueName: \"kubernetes.io/projected/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-kube-api-access-4s6tm\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-kfvjh\" (UID: \"142a47dc-0a88-4e03-b0b7-8a62d8b8e861\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:11:55.978800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.978715 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 06:11:55.979062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:55.979012 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 06:11:56.024645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.024593 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s6tm\" (UniqueName: \"kubernetes.io/projected/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-kube-api-access-4s6tm\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-kfvjh\" (UID: \"142a47dc-0a88-4e03-b0b7-8a62d8b8e861\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:11:56.027416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.026391 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-plugins-config-volume\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-kfvjh\" (UID: \"142a47dc-0a88-4e03-b0b7-8a62d8b8e861\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:11:56.027626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.027602 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-plugins-config-volume\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-kfvjh\" (UID: \"142a47dc-0a88-4e03-b0b7-8a62d8b8e861\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:11:56.091108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.091017 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjbrn\" (UniqueName: \"kubernetes.io/projected/dc51944b-1255-4681-a6dc-fc6e503116b5-kube-api-access-hjbrn\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9\" (UID: \"dc51944b-1255-4681-a6dc-fc6e503116b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:11:56.148515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.137995 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s6tm\" (UniqueName: \"kubernetes.io/projected/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-kube-api-access-4s6tm\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-kfvjh\" (UID: \"142a47dc-0a88-4e03-b0b7-8a62d8b8e861\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:11:56.186348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.171291 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 06:11:56.186348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.171382 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" podUID="1bcadd15-1201-491a-8348-dd39b94f9bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 06:11:56.186348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.171876 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:11:56.371335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.371056 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:11:56.414388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.401907 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:11:56.414388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.401963 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 06:11:56.870729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.870260 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 06:11:56.870729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:56.870348 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 06:11:57.139235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:57.139144 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 06:11:57.139235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:57.139214 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" podUID="1bcadd15-1201-491a-8348-dd39b94f9bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 06:11:57.312597 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:57.312557 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc51944b_1255_4681_a6dc_fc6e503116b5.slice/crio-2500263933beeacfd362b79ca96e4af42d3fcb09eb3ef7c0e80e62b3606bc3cc WatchSource:0}: Error finding container 2500263933beeacfd362b79ca96e4af42d3fcb09eb3ef7c0e80e62b3606bc3cc: Status 404 returned error can't find the container with id 2500263933beeacfd362b79ca96e4af42d3fcb09eb3ef7c0e80e62b3606bc3cc Oct 30 06:11:57.314083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:57.314058 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9"] Oct 30 06:11:57.319186 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:11:57.319157 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142a47dc_0a88_4e03_b0b7_8a62d8b8e861.slice/crio-ecc79018cc4dfb459160efa9ce60d4137a8e2c7871dfd21b76b7c7076ae301fb WatchSource:0}: Error finding container ecc79018cc4dfb459160efa9ce60d4137a8e2c7871dfd21b76b7c7076ae301fb: Status 404 returned error can't find the container with id ecc79018cc4dfb459160efa9ce60d4137a8e2c7871dfd21b76b7c7076ae301fb Oct 30 06:11:57.325095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:57.323237 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh"] Oct 30 06:11:57.395538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:57.390630 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:11:57.395538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:57.390716 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 06:11:57.873335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:57.871660 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 06:11:57.873335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:57.871724 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 06:11:58.196334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.193609 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 06:11:58.196334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.193670 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" podUID="1bcadd15-1201-491a-8348-dd39b94f9bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 06:11:58.233838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.233610 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" event={"ID":"dc51944b-1255-4681-a6dc-fc6e503116b5","Type":"ContainerStarted","Data":"38c120a9d7ea02f0a2c3d955a262bb04ea8fccafa61502284760c8520ca5f798"} Oct 30 06:11:58.233838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.233682 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:11:58.233838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.233710 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" event={"ID":"dc51944b-1255-4681-a6dc-fc6e503116b5","Type":"ContainerStarted","Data":"2500263933beeacfd362b79ca96e4af42d3fcb09eb3ef7c0e80e62b3606bc3cc"} Oct 30 06:11:58.234390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.234353 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" event={"ID":"142a47dc-0a88-4e03-b0b7-8a62d8b8e861","Type":"ContainerStarted","Data":"3c2b21d10b2791e311806d6a3c006ef85c358fb50a9b94adf80aaec1c7d33e7f"} Oct 30 06:11:58.234522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.234405 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" event={"ID":"142a47dc-0a88-4e03-b0b7-8a62d8b8e861","Type":"ContainerStarted","Data":"ecc79018cc4dfb459160efa9ce60d4137a8e2c7871dfd21b76b7c7076ae301fb"} Oct 30 06:11:58.234647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.234628 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:11:58.308193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.301076 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" podStartSLOduration=3.301041178 podStartE2EDuration="3.301041178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:58.27334887 +0000 UTC m=+1132.269305881" watchObservedRunningTime="2026-10-30 06:11:58.301041178 +0000 UTC m=+1132.296998188" Oct 30 06:11:58.318937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.318877 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" podStartSLOduration=3.318844256 podStartE2EDuration="3.318844256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:58.316100694 +0000 UTC m=+1132.312057705" watchObservedRunningTime="2026-10-30 06:11:58.318844256 +0000 UTC m=+1132.314801268" Oct 30 06:11:58.379329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.374436 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:11:58.379329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:58.374495 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 06:11:59.137615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:59.137584 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 06:11:59.137865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:59.137830 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" podUID="1bcadd15-1201-491a-8348-dd39b94f9bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 06:11:59.330602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:59.330506 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:59.331099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:59.331065 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:59.332541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:59.332512 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:11:59.547768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:59.547686 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:59.549003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:59.548977 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:11:59.569643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:11:59.569614 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:12:00.134965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:00.134928 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 06:12:00.135150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:00.134986 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" podUID="1bcadd15-1201-491a-8348-dd39b94f9bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 06:12:01.551611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:01.551578 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:12:01.552849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:01.552774 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:12:01.553771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:01.553751 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:12:04.138331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:04.137403 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:04.155156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:04.155113 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:04.236487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:04.236429 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:04.366452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:04.366402 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:04.649345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:04.648997 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:05.043545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:05.043435 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 06:12:05.138503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:05.138456 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 06:12:05.139742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:05.139714 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:12:05.238450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:05.238405 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 06:12:05.239272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:05.239251 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:12:05.447337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:05.441124 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:12:05.451424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:05.451355 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:05.694333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:05.690240 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:05.717397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:05.697228 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:12:06.064393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:06.052393 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:06.198332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:06.198244 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 06:12:06.251332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:06.247632 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 06:12:06.270808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:06.270771 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:12:06.297333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:06.274100 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 06:12:06.457333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:06.447466 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:06.617906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:06.617823 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6g2sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:12:06.670847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:06.668390 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:06.732504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:06.712174 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:07.177343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:07.175589 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:07.275331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:07.273968 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:07.275331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:07.275093 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:07.311333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:07.296085 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:07.311333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:07.297099 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:12:07.637332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:07.636382 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:07.637332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:07.637067 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:12:07.792532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:07.762862 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:12:07.792532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:07.778442 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 06:12:08.334833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:08.334792 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:08.364379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:08.364331 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 06:12:08.405477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:08.405422 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:08.501010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:08.500976 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:12:08.518562 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:08.518533 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:12:08.648318 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:08.642256 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 06:12:08.649411 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:12:08.649284 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:12:08.673678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:08.673646 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:12:08.747625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:08.747593 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 06:12:08.748957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:08.748936 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:12:09.254260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:09.254229 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:12:10.273831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:10.273803 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:12:11.070527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.070497 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7"] Oct 30 06:12:11.076128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.076104 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.109850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.109824 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7"] Oct 30 06:12:11.115648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.115616 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.115819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.115674 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.115819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.115750 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrbf5\" (UniqueName: \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-kube-api-access-xrbf5\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.115949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.115851 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.115949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.115939 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.116074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.116041 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.116074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.116060 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.116183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.116118 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.116251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.116228 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.217674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.217637 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.217674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.217676 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.217920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.217696 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.217920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.217712 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrbf5\" (UniqueName: \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-kube-api-access-xrbf5\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.217920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.217754 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.217920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.217812 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.218125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.217927 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.218233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.217956 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.218430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.218385 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.218519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.218426 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.218519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.218454 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.218519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.218512 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.218636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.218592 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.218775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.218753 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.220223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.220199 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.221142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.221118 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.241936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.241895 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrbf5\" (UniqueName: \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-kube-api-access-xrbf5\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.242155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.242134 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2ltk7\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.392579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.392530 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:11.790224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.780703 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7"] Oct 30 06:12:11.790224 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:12:11.781714 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cdd044_ecb6_485e_9b34_9d1362aa3bfa.slice/crio-74770bff3864ccd847e10ad6beb3d9ff873af2fa962265ab9e93fcf1052b8fa7 WatchSource:0}: Error finding container 74770bff3864ccd847e10ad6beb3d9ff873af2fa962265ab9e93fcf1052b8fa7: Status 404 returned error can't find the container with id 74770bff3864ccd847e10ad6beb3d9ff873af2fa962265ab9e93fcf1052b8fa7 Oct 30 06:12:11.790224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.784455 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:11.790224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.784560 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:11.790224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:11.784604 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:12.299591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.299556 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h"] Oct 30 06:12:12.312718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.312685 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.334812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.334781 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h"] Oct 30 06:12:12.341465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.340739 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1ec766-openshift-default-dockercfg-85lxf\"" Oct 30 06:12:12.459118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.459079 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-token\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.464493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.464455 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-podinfo\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.474335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.468906 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d29e78-7d62-4e08-b297-9bef59250b4c-istiod-ca-cert\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.526821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.526791 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgfpr\" (UniqueName: \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-kube-api-access-jgfpr\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.528261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.528236 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-credential-socket\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.554716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.543448 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-envoy\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.554716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.543560 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-data\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.554716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.543605 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-certs\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.554716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.543640 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-socket\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.607429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.607380 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" event={"ID":"32cdd044-ecb6-485e-9b34-9d1362aa3bfa","Type":"ContainerStarted","Data":"4bdd58861e0c87f1c1043205a2593cfeeafeb774ad0079d8c4cdcd78edad54a9"} Oct 30 06:12:12.607671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.607449 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" event={"ID":"32cdd044-ecb6-485e-9b34-9d1362aa3bfa","Type":"ContainerStarted","Data":"74770bff3864ccd847e10ad6beb3d9ff873af2fa962265ab9e93fcf1052b8fa7"} Oct 30 06:12:12.665332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.653584 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-data\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.665332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.653638 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-certs\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.665332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.653667 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-socket\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.665332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.653711 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-token\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.665332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.653773 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-podinfo\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.665332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.653797 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d29e78-7d62-4e08-b297-9bef59250b4c-istiod-ca-cert\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.665332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.653872 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgfpr\" (UniqueName: \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-kube-api-access-jgfpr\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.665332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.654014 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-credential-socket\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.665332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.654036 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-envoy\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.665332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.654881 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-data\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.665332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.655109 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-certs\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.705331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.682394 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d29e78-7d62-4e08-b297-9bef59250b4c-istiod-ca-cert\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.705331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.691043 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-credential-socket\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.706020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.705992 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-socket\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.730058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.725149 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-podinfo\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.732013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.731922 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-envoy\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.747285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.747252 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgfpr\" (UniqueName: \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-kube-api-access-jgfpr\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.767159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.767130 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-token\") pod \"e2e-api-a-1ec766-openshift-default-69f8887585-55l2h\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:12.948049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:12.948017 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:13.327745 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:12:13.327715 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d29e78_7d62_4e08_b297_9bef59250b4c.slice/crio-90dfca69954a20db909b99238e0d463078b8c6e553eb27eebd52ff1a471763a6 WatchSource:0}: Error finding container 90dfca69954a20db909b99238e0d463078b8c6e553eb27eebd52ff1a471763a6: Status 404 returned error can't find the container with id 90dfca69954a20db909b99238e0d463078b8c6e553eb27eebd52ff1a471763a6 Oct 30 06:12:13.331207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.331164 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:13.331346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.331261 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:13.331406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.331339 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:13.333259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.333197 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" podStartSLOduration=2.333180537 podStartE2EDuration="2.333180537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:12.705920815 +0000 UTC m=+1146.701877826" watchObservedRunningTime="2026-10-30 06:12:13.333180537 +0000 UTC m=+1147.329137548" Oct 30 06:12:13.334857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.334837 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h"] Oct 30 06:12:13.393235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.393208 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:13.394770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.394737 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2ltk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:12:13.394922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.394782 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 06:12:13.619255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.619213 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" event={"ID":"c0d29e78-7d62-4e08-b297-9bef59250b4c","Type":"ContainerStarted","Data":"17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1"} Oct 30 06:12:13.619748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.619289 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" event={"ID":"c0d29e78-7d62-4e08-b297-9bef59250b4c","Type":"ContainerStarted","Data":"90dfca69954a20db909b99238e0d463078b8c6e553eb27eebd52ff1a471763a6"} Oct 30 06:12:13.707749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.707684 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" podStartSLOduration=1.707665001 podStartE2EDuration="1.707665001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:13.680639005 +0000 UTC m=+1147.676596028" watchObservedRunningTime="2026-10-30 06:12:13.707665001 +0000 UTC m=+1147.703622010" Oct 30 06:12:13.948636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:13.948555 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:14.393999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:14.393965 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2ltk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:12:14.394199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:14.394017 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 06:12:14.974758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:14.974421 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:15.393880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:15.393769 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2ltk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:12:15.394081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:15.393829 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 06:12:15.637553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:15.637515 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:15.638613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:15.638585 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:12:16.401342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:16.400739 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2ltk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:12:16.401342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:16.400797 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 06:12:17.393957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:17.393915 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2ltk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:12:17.394179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:17.393981 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 06:12:18.410093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:18.410048 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2ltk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:12:18.410592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:18.410117 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 06:12:18.860381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:18.850385 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl"] Oct 30 06:12:18.860381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:18.857637 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:12:18.888616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:18.888584 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1ec766\"" Oct 30 06:12:18.939141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:18.939106 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl"] Oct 30 06:12:18.948667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:18.948581 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz69c\" (UniqueName: \"kubernetes.io/projected/3e720b79-23aa-4ff6-905e-fa74f201864d-kube-api-access-qz69c\") pod \"payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl\" (UID: \"3e720b79-23aa-4ff6-905e-fa74f201864d\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:12:18.949012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:18.948989 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e720b79-23aa-4ff6-905e-fa74f201864d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl\" (UID: \"3e720b79-23aa-4ff6-905e-fa74f201864d\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:12:18.966238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:18.964715 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1ec766-dockercfg-47w5p\"" Oct 30 06:12:19.058225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.058181 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz69c\" (UniqueName: \"kubernetes.io/projected/3e720b79-23aa-4ff6-905e-fa74f201864d-kube-api-access-qz69c\") pod \"payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl\" (UID: \"3e720b79-23aa-4ff6-905e-fa74f201864d\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:12:19.058597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.058573 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e720b79-23aa-4ff6-905e-fa74f201864d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl\" (UID: \"3e720b79-23aa-4ff6-905e-fa74f201864d\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:12:19.096137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.079257 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e720b79-23aa-4ff6-905e-fa74f201864d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl\" (UID: \"3e720b79-23aa-4ff6-905e-fa74f201864d\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:12:19.109948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.109924 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz69c\" (UniqueName: \"kubernetes.io/projected/3e720b79-23aa-4ff6-905e-fa74f201864d-kube-api-access-qz69c\") pod \"payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl\" (UID: \"3e720b79-23aa-4ff6-905e-fa74f201864d\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:12:19.248134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.248051 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:12:19.393019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.392970 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2ltk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:12:19.393171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.393036 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 06:12:19.654950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.654911 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl"] Oct 30 06:12:19.680343 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:12:19.680283 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e720b79_23aa_4ff6_905e_fa74f201864d.slice/crio-8e7f19d3842346ecdd353aacd4636d8e9b3e2aa51a2e4a833434c83b1a09e6be WatchSource:0}: Error finding container 8e7f19d3842346ecdd353aacd4636d8e9b3e2aa51a2e4a833434c83b1a09e6be: Status 404 returned error can't find the container with id 8e7f19d3842346ecdd353aacd4636d8e9b3e2aa51a2e4a833434c83b1a09e6be Oct 30 06:12:19.782376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.764091 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq"] Oct 30 06:12:19.809521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.809484 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq"] Oct 30 06:12:19.811191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.809861 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:19.825141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.824024 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60978a8f-5gbzq\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:19.825141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.824173 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60978a8f-5gbzq\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:19.825141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.824207 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d786\" (UniqueName: \"kubernetes.io/projected/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-kube-api-access-5d786\") pod \"maas-api-revoke-sub-60978a8f-5gbzq\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:19.828538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.828369 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:12:19.828677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.828558 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:12:19.828677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.828624 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:12:19.832383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.829274 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:12:19.846701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.846675 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" event={"ID":"3e720b79-23aa-4ff6-905e-fa74f201864d","Type":"ContainerStarted","Data":"6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce"} Oct 30 06:12:19.846896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.846883 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:12:19.847015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.847000 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" event={"ID":"3e720b79-23aa-4ff6-905e-fa74f201864d","Type":"ContainerStarted","Data":"8e7f19d3842346ecdd353aacd4636d8e9b3e2aa51a2e4a833434c83b1a09e6be"} Oct 30 06:12:19.925984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.925898 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60978a8f-5gbzq\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:19.925984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.925948 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d786\" (UniqueName: \"kubernetes.io/projected/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-kube-api-access-5d786\") pod \"maas-api-revoke-sub-60978a8f-5gbzq\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:19.926168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.926119 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60978a8f-5gbzq\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:19.926710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.926681 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60978a8f-5gbzq\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:19.928627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.928602 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60978a8f-5gbzq\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:19.951552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:19.951523 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d786\" (UniqueName: \"kubernetes.io/projected/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-kube-api-access-5d786\") pod \"maas-api-revoke-sub-60978a8f-5gbzq\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:20.139549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:20.139516 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:20.394400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:20.394272 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2ltk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:12:20.394400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:20.394354 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 06:12:20.509971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:20.509875 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" podStartSLOduration=2.509860541 podStartE2EDuration="2.509860541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:19.91218716 +0000 UTC m=+1153.908144170" watchObservedRunningTime="2026-10-30 06:12:20.509860541 +0000 UTC m=+1154.505817612" Oct 30 06:12:20.511966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:20.511939 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq"] Oct 30 06:12:20.514560 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:12:20.514532 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cb130b_ad14_4726_9a79_8317f4e9fbd3.slice/crio-e6e06d7cdc8e3a54b0fbd2f94ea31aefdf122137b8ca147639a9ecefa85cb462 WatchSource:0}: Error finding container e6e06d7cdc8e3a54b0fbd2f94ea31aefdf122137b8ca147639a9ecefa85cb462: Status 404 returned error can't find the container with id e6e06d7cdc8e3a54b0fbd2f94ea31aefdf122137b8ca147639a9ecefa85cb462 Oct 30 06:12:20.885145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:20.885104 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" event={"ID":"b2cb130b-ad14-4726-9a79-8317f4e9fbd3","Type":"ContainerStarted","Data":"e6e06d7cdc8e3a54b0fbd2f94ea31aefdf122137b8ca147639a9ecefa85cb462"} Oct 30 06:12:21.793348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:21.793270 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:21.794275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:21.794257 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:21.795141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:21.795111 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:12:21.894808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:21.894778 5001 generic.go:356] "Generic (PLEG): container finished" podID="b2cb130b-ad14-4726-9a79-8317f4e9fbd3" containerID="8ea22c447766dc60b288fae213ecd351082f825a50ccc97ef7739210d2fd75ef" exitCode=0 Oct 30 06:12:21.895209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:21.894845 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" event={"ID":"b2cb130b-ad14-4726-9a79-8317f4e9fbd3","Type":"ContainerDied","Data":"8ea22c447766dc60b288fae213ecd351082f825a50ccc97ef7739210d2fd75ef"} Oct 30 06:12:23.066789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.066760 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:23.129649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.129623 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-openshift-ca\") pod \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " Oct 30 06:12:23.129809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.129732 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-xks-ca\") pod \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " Oct 30 06:12:23.129964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.129943 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-kube-api-access-5d786\" (UniqueName: \"kubernetes.io/projected/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-kube-api-access-5d786\") pod \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\" (UID: \"b2cb130b-ad14-4726-9a79-8317f4e9fbd3\") " Oct 30 06:12:23.130044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.130026 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-openshift-ca" pod "b2cb130b-ad14-4726-9a79-8317f4e9fbd3" (UID: "b2cb130b-ad14-4726-9a79-8317f4e9fbd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:23.130468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.130443 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:12:23.132148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.132119 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-kube-api-access-5d786" pod "b2cb130b-ad14-4726-9a79-8317f4e9fbd3" (UID: "b2cb130b-ad14-4726-9a79-8317f4e9fbd3"). InnerVolumeSpecName "kube-api-access-5d786". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:23.132275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.132257 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-xks-ca" pod "b2cb130b-ad14-4726-9a79-8317f4e9fbd3" (UID: "b2cb130b-ad14-4726-9a79-8317f4e9fbd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:12:23.232044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.231950 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d786\" (UniqueName: \"kubernetes.io/projected/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-kube-api-access-5d786\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:12:23.232044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.231984 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2cb130b-ad14-4726-9a79-8317f4e9fbd3-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:12:23.461330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.460869 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:12:23.982326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.975101 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" event={"ID":"b2cb130b-ad14-4726-9a79-8317f4e9fbd3","Type":"ContainerDied","Data":"e6e06d7cdc8e3a54b0fbd2f94ea31aefdf122137b8ca147639a9ecefa85cb462"} Oct 30 06:12:23.982326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.975190 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6e06d7cdc8e3a54b0fbd2f94ea31aefdf122137b8ca147639a9ecefa85cb462" Oct 30 06:12:23.982326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:23.975288 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq" Oct 30 06:12:28.259339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:28.254817 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 06:12:28.315332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:28.314386 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:28.407330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:28.407109 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 06:12:28.534029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:28.533806 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:30.961869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:30.961780 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:12:36.262235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:36.262201 5001 generic.go:356] "Generic (PLEG): container finished" podID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerID="042d9c4c3e3598227335bc182d22baa9e8e698eef9278c5f3f985434feff7711" exitCode=0 Oct 30 06:12:36.262693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:36.262284 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" event={"ID":"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39","Type":"ContainerDied","Data":"042d9c4c3e3598227335bc182d22baa9e8e698eef9278c5f3f985434feff7711"} Oct 30 06:12:36.262736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:36.262700 5001 scope.go:122] "RemoveContainer" containerID="042d9c4c3e3598227335bc182d22baa9e8e698eef9278c5f3f985434feff7711" Oct 30 06:12:36.264148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:36.264122 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:36.264208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:36.264194 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:36.264242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:36.264224 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:36.720845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:36.720809 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:12:37.287554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.287519 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" event={"ID":"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39","Type":"ContainerStarted","Data":"4c7e72eeef02b3a136ec36ccbb5d2dd2c98e7aa4f836f56d6080899d86199529"} Oct 30 06:12:37.410346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.407801 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:12:37.410346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.410158 5001 patch_prober.go:29] interesting pod/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:12:37.410346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.410214 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 06:12:37.416328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.416283 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc"] Oct 30 06:12:37.417323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.417287 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b2cb130b-ad14-4726-9a79-8317f4e9fbd3" containerName="revoke-keys" Oct 30 06:12:37.417376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.417326 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cb130b-ad14-4726-9a79-8317f4e9fbd3" containerName="revoke-keys" Oct 30 06:12:37.417510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.417498 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cb130b-ad14-4726-9a79-8317f4e9fbd3" containerName="revoke-keys" Oct 30 06:12:37.422028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.422007 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.423289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.423262 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm"] Oct 30 06:12:37.427980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.427962 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.484331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.460911 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc"] Oct 30 06:12:37.491402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.490232 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.491582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.491450 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.491582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.491533 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j97zc\" (UniqueName: \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-kube-api-access-j97zc\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.491582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.491565 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74df9a78-380f-4540-a6b5-8bfefb73a099-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.491748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.491633 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.491748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.491658 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.491748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.491741 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.491898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.491786 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.491898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.491812 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.493821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.493803 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm"] Oct 30 06:12:37.593461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.593378 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtkmn\" (UniqueName: \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-kube-api-access-vtkmn\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.593461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.593415 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.593461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.593447 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j97zc\" (UniqueName: \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-kube-api-access-j97zc\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.593688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.593588 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74df9a78-380f-4540-a6b5-8bfefb73a099-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.593688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.593614 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fe3b62-36bb-4285-805c-b88ccef37319-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.593688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.593652 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.593688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.593680 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.593853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.593727 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.593853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.593836 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fe3b62-36bb-4285-805c-b88ccef37319-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.593982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.593960 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.594048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594020 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.594102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594046 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.594156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594130 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.594261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594241 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.594347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594281 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.594403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594376 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.594443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594414 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74df9a78-380f-4540-a6b5-8bfefb73a099-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.594443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594434 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.594511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594503 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.594555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594525 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.594593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594543 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.594593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594577 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.594922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.594898 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.596264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.596242 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.596966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.596946 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.620243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.620215 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.620474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.620456 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j97zc\" (UniqueName: \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-kube-api-access-j97zc\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.695975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.695943 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.695983 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.696070 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.696178 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.696254 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtkmn\" (UniqueName: \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-kube-api-access-vtkmn\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.696284 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.696357 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fe3b62-36bb-4285-805c-b88ccef37319-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.696417 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.696460 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fe3b62-36bb-4285-805c-b88ccef37319-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.696502 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.696570 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.696610 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.696933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.696914 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.697185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.697168 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fe3b62-36bb-4285-805c-b88ccef37319-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.698617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.698596 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.698859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.698843 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fe3b62-36bb-4285-805c-b88ccef37319-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.721266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.721238 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.721819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.721800 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtkmn\" (UniqueName: \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-kube-api-access-vtkmn\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:37.775861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.775576 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:37.800668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:37.800639 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:38.106962 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:12:38.106923 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74df9a78_380f_4540_a6b5_8bfefb73a099.slice/crio-edcffcd65124f3b61d613b1054cd18919e63d5289c6a49bf3ffe3c73d27acf16 WatchSource:0}: Error finding container edcffcd65124f3b61d613b1054cd18919e63d5289c6a49bf3ffe3c73d27acf16: Status 404 returned error can't find the container with id edcffcd65124f3b61d613b1054cd18919e63d5289c6a49bf3ffe3c73d27acf16 Oct 30 06:12:38.107256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.107234 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc"] Oct 30 06:12:38.109668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.109628 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:38.109783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.109714 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:38.109783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.109754 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:38.188060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.188027 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm"] Oct 30 06:12:38.188799 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:12:38.188768 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fe3b62_36bb_4285_805c_b88ccef37319.slice/crio-569d706458f63a4665c8681763dea5b0da4fce8caef2aaf89366ad7954a6decd WatchSource:0}: Error finding container 569d706458f63a4665c8681763dea5b0da4fce8caef2aaf89366ad7954a6decd: Status 404 returned error can't find the container with id 569d706458f63a4665c8681763dea5b0da4fce8caef2aaf89366ad7954a6decd Oct 30 06:12:38.191396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.191360 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:38.191469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.191435 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:38.191469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.191466 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:38.301641 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:12:38.301608 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:12:38.302852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.302829 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" event={"ID":"37fe3b62-36bb-4285-805c-b88ccef37319","Type":"ContainerStarted","Data":"569d706458f63a4665c8681763dea5b0da4fce8caef2aaf89366ad7954a6decd"} Oct 30 06:12:38.305515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.305491 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" event={"ID":"74df9a78-380f-4540-a6b5-8bfefb73a099","Type":"ContainerStarted","Data":"feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5"} Oct 30 06:12:38.305619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.305534 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" event={"ID":"74df9a78-380f-4540-a6b5-8bfefb73a099","Type":"ContainerStarted","Data":"edcffcd65124f3b61d613b1054cd18919e63d5289c6a49bf3ffe3c73d27acf16"} Oct 30 06:12:38.375028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.374968 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" podStartSLOduration=1.374948961 podStartE2EDuration="1.374948961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:38.36898832 +0000 UTC m=+1172.364945331" watchObservedRunningTime="2026-10-30 06:12:38.374948961 +0000 UTC m=+1172.370905988" Oct 30 06:12:38.407684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.407630 5001 patch_prober.go:29] interesting pod/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:12:38.407684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.407676 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 06:12:38.779401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:38.776381 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:39.437332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:39.435890 5001 patch_prober.go:29] interesting pod/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:12:39.437332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:39.435960 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 06:12:39.542562 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:39.542525 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" event={"ID":"37fe3b62-36bb-4285-805c-b88ccef37319","Type":"ContainerStarted","Data":"4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10"} Oct 30 06:12:39.642384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:39.636271 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" podStartSLOduration=2.636247446 podStartE2EDuration="2.636247446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:39.62175174 +0000 UTC m=+1173.617708793" watchObservedRunningTime="2026-10-30 06:12:39.636247446 +0000 UTC m=+1173.632204457" Oct 30 06:12:39.782394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:39.781558 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 06:12:39.782394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:39.781622 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 06:12:39.804330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:39.803290 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:39.804330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:39.803568 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:12:39.804330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:39.803632 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 06:12:40.420582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:40.420545 5001 patch_prober.go:29] interesting pod/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:12:40.421728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:40.421695 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 06:12:40.780372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:40.780257 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 06:12:40.780985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:40.780952 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 06:12:40.810799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:40.810769 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:12:40.811041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:40.811019 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 06:12:41.047591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:40.945099 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl"] Oct 30 06:12:41.060337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.059506 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl"] Oct 30 06:12:41.060337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.059649 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.272645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.263207 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.272645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.263295 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbcr6\" (UniqueName: \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-kube-api-access-nbcr6\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.272645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.263396 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.272645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.263483 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.272645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.263509 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7ca03e-63bc-4107-85aa-73852f18cf40-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.272645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.263565 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.272645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.263599 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.272645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.263636 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.272645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.263685 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.385661 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.385717 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.385754 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.385800 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.385910 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.385957 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbcr6\" (UniqueName: \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-kube-api-access-nbcr6\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.386040 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.386115 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.386139 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7ca03e-63bc-4107-85aa-73852f18cf40-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.387162 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7ca03e-63bc-4107-85aa-73852f18cf40-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.387485 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.412790 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.413054 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.413334 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.415474 5001 patch_prober.go:29] interesting pod/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:12:41.440524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.415519 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 06:12:41.442121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.441973 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.450330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.443786 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbcr6\" (UniqueName: \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-kube-api-access-nbcr6\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.488328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.487169 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.491619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.491590 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.742971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.742853 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:41.802915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.802491 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:12:41.802915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.802549 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 06:12:41.804210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.804190 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 06:12:41.804394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:41.804374 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 06:12:42.424353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:42.418427 5001 patch_prober.go:29] interesting pod/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:12:42.424353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:42.418485 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 06:12:42.727634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:42.727521 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl"] Oct 30 06:12:42.801755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:42.801706 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:12:42.811420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:42.804217 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 06:12:42.839639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:42.824416 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 06:12:42.839639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:42.824471 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 06:12:42.839639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:42.839066 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:42.839639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:42.839177 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:42.839639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:42.839235 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:43.417713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:43.417679 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:12:43.419453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:43.419424 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:12:43.420476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:43.420452 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:12:43.796440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:43.781872 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 06:12:43.796440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:43.781925 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 06:12:43.836819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:43.829901 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:12:43.836819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:43.829956 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 06:12:43.961348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:43.960471 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" event={"ID":"1c7ca03e-63bc-4107-85aa-73852f18cf40","Type":"ContainerStarted","Data":"2ed43fa7a8dc76c8ba309717ded505c1ba20653e60821f6ffffcecd849e6f6c8"} Oct 30 06:12:43.968634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:43.968601 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" event={"ID":"1c7ca03e-63bc-4107-85aa-73852f18cf40","Type":"ContainerStarted","Data":"c9942f2bcacdbd3fc214c1b16f177f57a3b6f2686d9d359ae1ecbdb1f58a5565"} Oct 30 06:12:44.043183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:44.043117 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" podStartSLOduration=4.043095823 podStartE2EDuration="4.043095823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:44.033930819 +0000 UTC m=+1178.029887830" watchObservedRunningTime="2026-10-30 06:12:44.043095823 +0000 UTC m=+1178.039052834" Oct 30 06:12:44.750333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:44.745608 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:44.750333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:44.747150 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:12:44.750333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:44.747202 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" podUID="1c7ca03e-63bc-4107-85aa-73852f18cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 06:12:44.778332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:44.777702 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 06:12:44.778332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:44.777761 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 06:12:44.811332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:44.810726 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:12:44.811332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:44.810783 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 06:12:45.744072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:45.743983 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:12:45.744072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:45.744050 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" podUID="1c7ca03e-63bc-4107-85aa-73852f18cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 06:12:45.788455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:45.786691 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 06:12:45.788455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:45.786748 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 06:12:45.801128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:45.801042 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:12:45.801128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:45.801087 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 06:12:46.744326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:46.744261 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:12:46.744809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:46.744376 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" podUID="1c7ca03e-63bc-4107-85aa-73852f18cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 06:12:46.903916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:46.903885 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:46.904903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:46.904825 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:46.905874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:46.905856 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:12:46.929626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:46.929595 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:46.930506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:46.930472 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:46.931700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:46.931670 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:12:47.744376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:47.744348 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:12:47.744789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:47.744392 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" podUID="1c7ca03e-63bc-4107-85aa-73852f18cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 06:12:48.744716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:48.744684 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:12:48.745219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:48.744744 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" podUID="1c7ca03e-63bc-4107-85aa-73852f18cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 06:12:49.744400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:49.744362 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:12:49.744583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:49.744419 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" podUID="1c7ca03e-63bc-4107-85aa-73852f18cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 06:12:51.051077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:51.051042 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:51.051572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:51.051558 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:51.052457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:51.052430 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:12:52.991327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:52.987432 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n"] Oct 30 06:12:53.055041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.055007 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.072797 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.072765 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f789d7-openshift-default-dockercfg-wlrff\"" Oct 30 06:12:53.113333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.109233 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n"] Oct 30 06:12:53.221794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.221755 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-credential-socket\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.258762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.258663 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-data\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.266629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.266593 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-certs\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.273352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.266849 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-podinfo\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.273352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.266935 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istiod-ca-cert\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.273352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.267009 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-envoy\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.273352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.267050 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-token\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.273352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.267104 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-socket\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.273352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.267146 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9bnc\" (UniqueName: \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-kube-api-access-t9bnc\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.379432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.379394 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-socket\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.466666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.466612 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bnc\" (UniqueName: \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-kube-api-access-t9bnc\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.485032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.484995 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-credential-socket\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.506452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.503828 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-data\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.506452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.504030 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-certs\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.506452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.504063 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-podinfo\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.506452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.504111 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istiod-ca-cert\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.506452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.504192 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-envoy\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.506452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.504230 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-token\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.506452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.388586 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-socket\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.560976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.560900 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-data\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.561921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.561468 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-certs\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.568101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.568069 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istiod-ca-cert\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.569402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.569373 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-podinfo\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.570156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.570135 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-credential-socket\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.572255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.572206 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-envoy\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.576264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.576209 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bnc\" (UniqueName: \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-kube-api-access-t9bnc\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.595936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.595896 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-token\") pod \"e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:53.873635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:53.873593 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:54.214405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:54.214372 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n"] Oct 30 06:12:54.215082 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:12:54.215055 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43aeac6_adbb_433c_92b2_3d92cb5c9660.slice/crio-7aba70999081b924e3382f1059d6116b3ed39dbcd5699b27dd1df9c841819fc4 WatchSource:0}: Error finding container 7aba70999081b924e3382f1059d6116b3ed39dbcd5699b27dd1df9c841819fc4: Status 404 returned error can't find the container with id 7aba70999081b924e3382f1059d6116b3ed39dbcd5699b27dd1df9c841819fc4 Oct 30 06:12:54.217610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:54.217568 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:54.217697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:54.217654 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:54.217739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:54.217697 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:54.375326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:54.369540 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" event={"ID":"f43aeac6-adbb-433c-92b2-3d92cb5c9660","Type":"ContainerStarted","Data":"7aba70999081b924e3382f1059d6116b3ed39dbcd5699b27dd1df9c841819fc4"} Oct 30 06:12:55.388697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.388654 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" event={"ID":"f43aeac6-adbb-433c-92b2-3d92cb5c9660","Type":"ContainerStarted","Data":"e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a"} Oct 30 06:12:55.486393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.486338 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" podStartSLOduration=3.486317885 podStartE2EDuration="3.486317885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:55.48175123 +0000 UTC m=+1189.477708244" watchObservedRunningTime="2026-10-30 06:12:55.486317885 +0000 UTC m=+1189.482274892" Oct 30 06:12:55.882670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.882615 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:55.888281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.888252 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j"] Oct 30 06:12:55.893278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.893255 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:55.893439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.893364 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:55.918487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.918456 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j"] Oct 30 06:12:55.979332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.975244 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd10d8d-70a5-4897-81c8-0875ff56a289-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:55.979332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.975332 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:55.979332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.975386 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:55.979332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.975427 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zp64\" (UniqueName: \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-kube-api-access-8zp64\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:55.979332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.975547 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:55.979332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.975574 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:55.979332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.975618 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:55.979332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.975650 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:55.979332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:55.975719 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.078507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.078472 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.079221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.079194 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.079402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.079132 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.079537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.079396 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.079664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.079564 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.079664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.079627 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.079801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.079782 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd10d8d-70a5-4897-81c8-0875ff56a289-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.079971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.079953 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.080118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.080102 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.080249 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.080233 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zp64\" (UniqueName: \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-kube-api-access-8zp64\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.081009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.080015 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.081298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.081277 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.081685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.081666 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.082200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.082180 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd10d8d-70a5-4897-81c8-0875ff56a289-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.083255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.083025 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.084683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.084663 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.104437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.104410 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.104619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.104599 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zp64\" (UniqueName: \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-kube-api-access-8zp64\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.209138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.209054 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:56.452149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.452094 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:56.454345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.454139 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:12:56.714612 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:12:56.714575 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd10d8d_70a5_4897_81c8_0875ff56a289.slice/crio-3618d78b3cb2430504b681fe5b663e47c5c22cdcc8a363093851a724fce01e9c WatchSource:0}: Error finding container 3618d78b3cb2430504b681fe5b663e47c5c22cdcc8a363093851a724fce01e9c: Status 404 returned error can't find the container with id 3618d78b3cb2430504b681fe5b663e47c5c22cdcc8a363093851a724fce01e9c Oct 30 06:12:56.714742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.714649 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j"] Oct 30 06:12:56.728405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.727530 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:56.728405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.727676 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:56.728405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:56.727785 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:57.492333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:57.488458 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" event={"ID":"abd10d8d-70a5-4897-81c8-0875ff56a289","Type":"ContainerStarted","Data":"93d6986eb5bb60ad129b6a6be3d0c7c16a9fe3dc34d4a1991056ce72c46519c6"} Oct 30 06:12:57.492333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:57.488518 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" event={"ID":"abd10d8d-70a5-4897-81c8-0875ff56a289","Type":"ContainerStarted","Data":"3618d78b3cb2430504b681fe5b663e47c5c22cdcc8a363093851a724fce01e9c"} Oct 30 06:12:58.210333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:58.210187 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:12:58.211804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:58.211779 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 06:12:58.212047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:58.212018 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" podUID="abd10d8d-70a5-4897-81c8-0875ff56a289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 06:12:59.210345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:59.210097 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 06:12:59.210345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:12:59.210143 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" podUID="abd10d8d-70a5-4897-81c8-0875ff56a289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 06:13:00.109393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.109328 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" podStartSLOduration=5.109283663 podStartE2EDuration="5.109283663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:57.540086733 +0000 UTC m=+1191.536043743" watchObservedRunningTime="2026-10-30 06:13:00.109283663 +0000 UTC m=+1194.105240681" Oct 30 06:13:00.123563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.123525 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j"] Oct 30 06:13:00.130179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.130151 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:13:00.136173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.136143 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j"] Oct 30 06:13:00.183893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.183574 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f789d7-dockercfg-vczdf\"" Oct 30 06:13:00.183893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.183767 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f789d7\"" Oct 30 06:13:00.214828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.214331 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 06:13:00.214828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.214414 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" podUID="abd10d8d-70a5-4897-81c8-0875ff56a289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 06:13:00.231902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.231793 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62q6s\" (UniqueName: \"kubernetes.io/projected/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-kube-api-access-62q6s\") pod \"payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j\" (UID: \"ddbaa0c1-73a6-473a-89e6-79fb010aa88d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:13:00.232857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.232156 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j\" (UID: \"ddbaa0c1-73a6-473a-89e6-79fb010aa88d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:13:00.390527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.381724 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62q6s\" (UniqueName: \"kubernetes.io/projected/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-kube-api-access-62q6s\") pod \"payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j\" (UID: \"ddbaa0c1-73a6-473a-89e6-79fb010aa88d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:13:00.402336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.390809 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j\" (UID: \"ddbaa0c1-73a6-473a-89e6-79fb010aa88d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:13:00.426875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.403854 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j\" (UID: \"ddbaa0c1-73a6-473a-89e6-79fb010aa88d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:13:00.526165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.526093 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62q6s\" (UniqueName: \"kubernetes.io/projected/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-kube-api-access-62q6s\") pod \"payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j\" (UID: \"ddbaa0c1-73a6-473a-89e6-79fb010aa88d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:13:00.537661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.537629 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8"] Oct 30 06:13:00.614335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.613659 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:13:00.723430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.640171 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8"] Oct 30 06:13:00.723430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.710777 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4175ac21-794a-4a88-9305-926195379728-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8\" (UID: \"4175ac21-794a-4a88-9305-926195379728\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:13:00.723430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.710938 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdzv\" (UniqueName: \"kubernetes.io/projected/4175ac21-794a-4a88-9305-926195379728-kube-api-access-6zdzv\") pod \"payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8\" (UID: \"4175ac21-794a-4a88-9305-926195379728\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:13:00.798731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.796658 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:13:00.844171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.844081 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdzv\" (UniqueName: \"kubernetes.io/projected/4175ac21-794a-4a88-9305-926195379728-kube-api-access-6zdzv\") pod \"payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8\" (UID: \"4175ac21-794a-4a88-9305-926195379728\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:13:00.844972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.844942 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4175ac21-794a-4a88-9305-926195379728-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8\" (UID: \"4175ac21-794a-4a88-9305-926195379728\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:13:00.914400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.914368 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdzv\" (UniqueName: \"kubernetes.io/projected/4175ac21-794a-4a88-9305-926195379728-kube-api-access-6zdzv\") pod \"payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8\" (UID: \"4175ac21-794a-4a88-9305-926195379728\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:13:00.915077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:00.914911 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4175ac21-794a-4a88-9305-926195379728-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8\" (UID: \"4175ac21-794a-4a88-9305-926195379728\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:13:01.009198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.009083 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp"] Oct 30 06:13:01.019683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.019659 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:01.025511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.025259 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp"] Oct 30 06:13:01.038862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.038010 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:13:01.038862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.038729 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:13:01.083813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.069692 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:13:01.083813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.069762 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:13:01.150685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.150593 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-501ef623-hdrjp\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:01.150863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.150761 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-501ef623-hdrjp\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:01.151049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.151026 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9fbv\" (UniqueName: \"kubernetes.io/projected/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-kube-api-access-g9fbv\") pod \"maas-api-revoke-sub-501ef623-hdrjp\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:01.210081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.210012 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:13:01.211109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.211078 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 06:13:01.211228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.211138 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" podUID="abd10d8d-70a5-4897-81c8-0875ff56a289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 06:13:01.258124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.256257 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-501ef623-hdrjp\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:01.258124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.256328 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-501ef623-hdrjp\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:01.258124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.256689 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9fbv\" (UniqueName: \"kubernetes.io/projected/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-kube-api-access-g9fbv\") pod \"maas-api-revoke-sub-501ef623-hdrjp\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:01.258124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.257759 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-501ef623-hdrjp\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:01.262497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.262418 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-501ef623-hdrjp\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:01.289632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.289584 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j"] Oct 30 06:13:01.290758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.290622 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9fbv\" (UniqueName: \"kubernetes.io/projected/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-kube-api-access-g9fbv\") pod \"maas-api-revoke-sub-501ef623-hdrjp\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:01.299235 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:13:01.299195 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbaa0c1_73a6_473a_89e6_79fb010aa88d.slice/crio-8230490debc6cce3cabc8b0b83757b84399b1167db955541ac43e7a3d751c10a WatchSource:0}: Error finding container 8230490debc6cce3cabc8b0b83757b84399b1167db955541ac43e7a3d751c10a: Status 404 returned error can't find the container with id 8230490debc6cce3cabc8b0b83757b84399b1167db955541ac43e7a3d751c10a Oct 30 06:13:01.395676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.394534 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:01.590683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.589189 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8"] Oct 30 06:13:01.718482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.718447 5001 patch_prober.go:29] interesting pod/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:13:01.718640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.718511 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:13:01.751249 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.751215 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp"] Oct 30 06:13:01.789351 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:13:01.787150 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb750e5f_87e7_4a1b_a9f6_5da3ccaaad59.slice/crio-bda542d5b6e5fc0ac8ec8069d12e403612330e506d77e79d747ef25105dbc2de WatchSource:0}: Error finding container bda542d5b6e5fc0ac8ec8069d12e403612330e506d77e79d747ef25105dbc2de: Status 404 returned error can't find the container with id bda542d5b6e5fc0ac8ec8069d12e403612330e506d77e79d747ef25105dbc2de Oct 30 06:13:01.886752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.886711 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" event={"ID":"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59","Type":"ContainerStarted","Data":"bda542d5b6e5fc0ac8ec8069d12e403612330e506d77e79d747ef25105dbc2de"} Oct 30 06:13:01.888726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.888699 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" event={"ID":"ddbaa0c1-73a6-473a-89e6-79fb010aa88d","Type":"ContainerStarted","Data":"2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98"} Oct 30 06:13:01.888875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.888743 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" event={"ID":"ddbaa0c1-73a6-473a-89e6-79fb010aa88d","Type":"ContainerStarted","Data":"8230490debc6cce3cabc8b0b83757b84399b1167db955541ac43e7a3d751c10a"} Oct 30 06:13:01.888875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.888801 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:13:01.890620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.890592 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" event={"ID":"4175ac21-794a-4a88-9305-926195379728","Type":"ContainerStarted","Data":"b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e"} Oct 30 06:13:01.890699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.890644 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" event={"ID":"4175ac21-794a-4a88-9305-926195379728","Type":"ContainerStarted","Data":"a8772a35c59a5c9481dfe53275242885394ee867245786fd8673deeb8f0538a2"} Oct 30 06:13:01.890699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.890665 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:13:01.935637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.935588 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" podStartSLOduration=1.935574811 podStartE2EDuration="1.935574811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:01.930488173 +0000 UTC m=+1195.926445187" watchObservedRunningTime="2026-10-30 06:13:01.935574811 +0000 UTC m=+1195.931531821" Oct 30 06:13:01.981092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:01.981026 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" podStartSLOduration=1.981002985 podStartE2EDuration="1.981002985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:01.972551646 +0000 UTC m=+1195.968508656" watchObservedRunningTime="2026-10-30 06:13:01.981002985 +0000 UTC m=+1195.976959996" Oct 30 06:13:02.209825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:02.209737 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 06:13:02.211486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:02.211274 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" podUID="abd10d8d-70a5-4897-81c8-0875ff56a289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 06:13:02.906745 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:02.906636 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" event={"ID":"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59","Type":"ContainerStarted","Data":"fd23e40793b224b87684f205673c0d20ae02e86104eef6a9e03be7966429c8a2"} Oct 30 06:13:02.950925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:02.950869 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" podStartSLOduration=2.950855147 podStartE2EDuration="2.950855147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:02.948977643 +0000 UTC m=+1196.944934665" watchObservedRunningTime="2026-10-30 06:13:02.950855147 +0000 UTC m=+1196.946812184" Oct 30 06:13:03.210178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:03.210095 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 06:13:03.210178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:03.210150 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" podUID="abd10d8d-70a5-4897-81c8-0875ff56a289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 06:13:03.919609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:03.919567 5001 generic.go:356] "Generic (PLEG): container finished" podID="cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59" containerID="fd23e40793b224b87684f205673c0d20ae02e86104eef6a9e03be7966429c8a2" exitCode=0 Oct 30 06:13:03.920123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:03.919620 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" event={"ID":"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59","Type":"ContainerDied","Data":"fd23e40793b224b87684f205673c0d20ae02e86104eef6a9e03be7966429c8a2"} Oct 30 06:13:04.826760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:04.826731 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:13:04.827360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:04.827338 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:13:04.828324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:04.828288 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:13:05.163456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.163380 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:05.273168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.273133 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-openshift-ca\") pod \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " Oct 30 06:13:05.273388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.273227 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-xks-ca\") pod \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " Oct 30 06:13:05.273521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.273502 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-kube-api-access-g9fbv\" (UniqueName: \"kubernetes.io/projected/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-kube-api-access-g9fbv\") pod \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\" (UID: \"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59\") " Oct 30 06:13:05.274868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.274838 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-openshift-ca" pod "cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59" (UID: "cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:05.277064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.277041 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-xks-ca" pod "cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59" (UID: "cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:13:05.278418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.278398 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-kube-api-access-g9fbv" pod "cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59" (UID: "cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59"). InnerVolumeSpecName "kube-api-access-g9fbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:05.375720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.375685 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9fbv\" (UniqueName: \"kubernetes.io/projected/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-kube-api-access-g9fbv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:05.375720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.375716 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:05.375938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.375728 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:05.940049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.940021 5001 generic.go:356] "Generic (PLEG): container finished" podID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerID="17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1" exitCode=0 Oct 30 06:13:05.940225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.940097 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" event={"ID":"c0d29e78-7d62-4e08-b297-9bef59250b4c","Type":"ContainerDied","Data":"17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1"} Oct 30 06:13:05.940613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.940596 5001 scope.go:122] "RemoveContainer" containerID="17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1" Oct 30 06:13:05.941949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.941713 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" Oct 30 06:13:05.941949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.941746 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp" event={"ID":"cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59","Type":"ContainerDied","Data":"bda542d5b6e5fc0ac8ec8069d12e403612330e506d77e79d747ef25105dbc2de"} Oct 30 06:13:05.941949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.941782 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bda542d5b6e5fc0ac8ec8069d12e403612330e506d77e79d747ef25105dbc2de" Oct 30 06:13:05.942460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.942429 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:05.942545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.942499 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:05.942545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:05.942540 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:06.316454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:06.316368 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:13:06.326715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:06.326690 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:13:06.716159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:06.716124 5001 scope.go:122] "RemoveContainer" containerID="3d987c743ca510a4bada3ba201df18890881be47a6eb53b6771cd6903a8d67bc" Oct 30 06:13:06.726633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:06.726612 5001 scope.go:122] "RemoveContainer" containerID="e7c4ba65db23976950e691fb3f3cc9bfb5ebd92beff0f1ccd542c2c6e08207f7" Oct 30 06:13:06.737071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:06.737051 5001 scope.go:122] "RemoveContainer" containerID="e16c706a4ee36097b034325fbe8d1e7c5c129fe3a0fa0439314566c60acf3916" Oct 30 06:13:06.982394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:06.982210 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" event={"ID":"c0d29e78-7d62-4e08-b297-9bef59250b4c","Type":"ContainerStarted","Data":"39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4"} Oct 30 06:13:07.948945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:07.948918 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:13:07.949507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:07.949084 5001 patch_prober.go:29] interesting pod/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 06:13:07.949507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:07.949138 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 06:13:08.301217 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:13:08.301132 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:13:08.447252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:08.447210 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:13:08.957333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:08.950669 5001 patch_prober.go:29] interesting pod/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 06:13:08.957333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:08.950732 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 06:13:09.951030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:09.950992 5001 patch_prober.go:29] interesting pod/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 06:13:09.951513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:09.951469 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 06:13:10.650333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:10.650081 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 06:13:10.957383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:10.956693 5001 patch_prober.go:29] interesting pod/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 06:13:10.957383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:10.956756 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 06:13:11.409384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.409349 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc"] Oct 30 06:13:11.422902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.422863 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59" containerName="revoke-keys" Oct 30 06:13:11.423147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.423127 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59" containerName="revoke-keys" Oct 30 06:13:11.423536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.423517 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59" containerName="revoke-keys" Oct 30 06:13:11.438325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.428824 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:11.479381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.466946 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:13:11.484344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.480462 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:13:11.484344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.480629 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:13:11.484344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.480736 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:13:11.516358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.497436 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d086f215-gbvqc\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:11.516358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.497791 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-964j9\" (UniqueName: \"kubernetes.io/projected/8d553e23-eecd-4158-ad64-6470e751d604-kube-api-access-964j9\") pod \"maas-api-revoke-sub-d086f215-gbvqc\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:11.516358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.497927 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d086f215-gbvqc\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:11.557113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.557080 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc"] Oct 30 06:13:11.651337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.624526 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-964j9\" (UniqueName: \"kubernetes.io/projected/8d553e23-eecd-4158-ad64-6470e751d604-kube-api-access-964j9\") pod \"maas-api-revoke-sub-d086f215-gbvqc\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:11.702711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.702613 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d086f215-gbvqc\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:11.703124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.703098 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d086f215-gbvqc\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:11.747329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.704082 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d086f215-gbvqc\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:11.762132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.761416 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:13:11.834694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.834657 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 06:13:11.876678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.872983 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d086f215-gbvqc\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:11.876678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:11.873657 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-964j9\" (UniqueName: \"kubernetes.io/projected/8d553e23-eecd-4158-ad64-6470e751d604-kube-api-access-964j9\") pod \"maas-api-revoke-sub-d086f215-gbvqc\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:12.188382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:12.150592 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:12.188382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:12.151489 5001 patch_prober.go:29] interesting pod/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 06:13:12.188382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:12.151551 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 06:13:12.714485 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:12.714453 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc"] Oct 30 06:13:12.910019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:12.909989 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:13:12.910362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:12.910297 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:13:12.948606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:12.948576 5001 patch_prober.go:29] interesting pod/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 06:13:12.948794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:12.948635 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 06:13:13.299336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:13.299233 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" event={"ID":"8d553e23-eecd-4158-ad64-6470e751d604","Type":"ContainerStarted","Data":"9c2a7f92cd843105b66f457a01beec73dd17a8409fb7ecc469bb74a91632c10a"} Oct 30 06:13:13.299336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:13.299290 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" event={"ID":"8d553e23-eecd-4158-ad64-6470e751d604","Type":"ContainerStarted","Data":"71be2d12ad5d8ad3106350c3b9be3451281c155bf76243db1a71cb0bfe78b390"} Oct 30 06:13:13.345127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:13.345079 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" podStartSLOduration=2.34506649 podStartE2EDuration="2.34506649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:13.340823315 +0000 UTC m=+1207.336780320" watchObservedRunningTime="2026-10-30 06:13:13.34506649 +0000 UTC m=+1207.341023500" Oct 30 06:13:13.953853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:13.953821 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:13:13.954379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:13.954365 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:13:13.955057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:13.955040 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:13:14.314651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:14.314618 5001 generic.go:356] "Generic (PLEG): container finished" podID="8d553e23-eecd-4158-ad64-6470e751d604" containerID="9c2a7f92cd843105b66f457a01beec73dd17a8409fb7ecc469bb74a91632c10a" exitCode=0 Oct 30 06:13:14.314848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:14.314697 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" event={"ID":"8d553e23-eecd-4158-ad64-6470e751d604","Type":"ContainerDied","Data":"9c2a7f92cd843105b66f457a01beec73dd17a8409fb7ecc469bb74a91632c10a"} Oct 30 06:13:15.538693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:15.538664 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:15.566046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:15.566015 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d553e23-eecd-4158-ad64-6470e751d604-kube-api-access-964j9\" (UniqueName: \"kubernetes.io/projected/8d553e23-eecd-4158-ad64-6470e751d604-kube-api-access-964j9\") pod \"8d553e23-eecd-4158-ad64-6470e751d604\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " Oct 30 06:13:15.566205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:15.566134 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-xks-ca\") pod \"8d553e23-eecd-4158-ad64-6470e751d604\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " Oct 30 06:13:15.566351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:15.566295 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-openshift-ca\") pod \"8d553e23-eecd-4158-ad64-6470e751d604\" (UID: \"8d553e23-eecd-4158-ad64-6470e751d604\") " Oct 30 06:13:15.566655 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:15.566628 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-openshift-ca" pod "8d553e23-eecd-4158-ad64-6470e751d604" (UID: "8d553e23-eecd-4158-ad64-6470e751d604"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:15.566955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:15.566937 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:15.568572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:15.568548 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-xks-ca" pod "8d553e23-eecd-4158-ad64-6470e751d604" (UID: "8d553e23-eecd-4158-ad64-6470e751d604"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:13:15.568831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:15.568812 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d553e23-eecd-4158-ad64-6470e751d604-kube-api-access-964j9" pod "8d553e23-eecd-4158-ad64-6470e751d604" (UID: "8d553e23-eecd-4158-ad64-6470e751d604"). InnerVolumeSpecName "kube-api-access-964j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:15.668379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:15.668247 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d553e23-eecd-4158-ad64-6470e751d604-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:15.668379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:15.668278 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-964j9\" (UniqueName: \"kubernetes.io/projected/8d553e23-eecd-4158-ad64-6470e751d604-kube-api-access-964j9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:16.373196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:16.373168 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" Oct 30 06:13:16.373916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:16.373166 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc" event={"ID":"8d553e23-eecd-4158-ad64-6470e751d604","Type":"ContainerDied","Data":"71be2d12ad5d8ad3106350c3b9be3451281c155bf76243db1a71cb0bfe78b390"} Oct 30 06:13:16.373988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:16.373947 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71be2d12ad5d8ad3106350c3b9be3451281c155bf76243db1a71cb0bfe78b390" Oct 30 06:13:23.433104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:23.433070 5001 generic.go:356] "Generic (PLEG): container finished" podID="ecabf567-9ebd-4831-ad26-6ad20065fdbf" containerID="42832c55ea98d61eae218eda97c70578a13370bad70f89d627e01f5e21b701f0" exitCode=0 Oct 30 06:13:23.433507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:23.433120 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" event={"ID":"ecabf567-9ebd-4831-ad26-6ad20065fdbf","Type":"ContainerDied","Data":"42832c55ea98d61eae218eda97c70578a13370bad70f89d627e01f5e21b701f0"} Oct 30 06:13:24.002001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.001962 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl"] Oct 30 06:13:24.002896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.002873 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8d553e23-eecd-4158-ad64-6470e751d604" containerName="revoke-keys" Oct 30 06:13:24.002896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.002896 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d553e23-eecd-4158-ad64-6470e751d604" containerName="revoke-keys" Oct 30 06:13:24.003060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.003032 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d553e23-eecd-4158-ad64-6470e751d604" containerName="revoke-keys" Oct 30 06:13:24.005925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.005906 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.029371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.029344 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl"] Oct 30 06:13:24.060327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.053591 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e6df62-openshift-default-dockercfg-9d68h\"" Oct 30 06:13:24.095153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.095115 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-data\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.095411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.095265 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.095480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.095396 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-token\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.095578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.095556 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-credential-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.095645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.095593 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-envoy\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.095717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.095691 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g8x7\" (UniqueName: \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-kube-api-access-9g8x7\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.095775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.095735 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-certs\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.095775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.095768 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-podinfo\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.095981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.095962 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67c14bb-f110-45db-b9d8-4800a84b7277-istiod-ca-cert\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.217332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.216290 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67c14bb-f110-45db-b9d8-4800a84b7277-istiod-ca-cert\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.217332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.216403 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-data\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.217332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.216503 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.217332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.216556 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-token\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.217332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.216711 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-credential-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.217332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.216736 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-envoy\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.217332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.216787 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g8x7\" (UniqueName: \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-kube-api-access-9g8x7\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.217332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.216818 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-certs\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.217332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.216856 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-podinfo\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.221614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.221586 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67c14bb-f110-45db-b9d8-4800a84b7277-istiod-ca-cert\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.247596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.222661 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.247785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.223114 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-data\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.247886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.225036 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-credential-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.247981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.246835 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-certs\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.261358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.260715 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-envoy\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.278111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.278058 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-podinfo\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.278744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.278713 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g8x7\" (UniqueName: \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-kube-api-access-9g8x7\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.289164 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.289107 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-token\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.326993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.326729 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:24.808860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.808825 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65"] Oct 30 06:13:24.813300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.813278 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:24.829020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.828993 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:13:24.831936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.831901 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65"] Oct 30 06:13:24.836251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.836226 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:13:24.836400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.836262 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:13:24.836488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.836466 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:13:24.837451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.837430 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:13:24.947817 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:13:24.947791 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67c14bb_f110_45db_b9d8_4800a84b7277.slice/crio-85ebe40318517ed8856608b17249d8070a43eb75c4dd403eae35a30d2ca94af4 WatchSource:0}: Error finding container 85ebe40318517ed8856608b17249d8070a43eb75c4dd403eae35a30d2ca94af4: Status 404 returned error can't find the container with id 85ebe40318517ed8856608b17249d8070a43eb75c4dd403eae35a30d2ca94af4 Oct 30 06:13:24.949208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.949191 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecabf567-9ebd-4831-ad26-6ad20065fdbf-kube-api-access-8t287\" (UniqueName: \"kubernetes.io/projected/ecabf567-9ebd-4831-ad26-6ad20065fdbf-kube-api-access-8t287\") pod \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " Oct 30 06:13:24.949327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.949237 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-trusted-ca-bundle\") pod \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " Oct 30 06:13:24.949381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.949331 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl"] Oct 30 06:13:24.949506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.949487 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-service-ca-bundle\") pod \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " Oct 30 06:13:24.949668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.949639 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-trusted-ca-bundle" pod "ecabf567-9ebd-4831-ad26-6ad20065fdbf" (UID: "ecabf567-9ebd-4831-ad26-6ad20065fdbf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:24.949668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.949657 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecabf567-9ebd-4831-ad26-6ad20065fdbf-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ecabf567-9ebd-4831-ad26-6ad20065fdbf-archives-path\") pod \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\" (UID: \"ecabf567-9ebd-4831-ad26-6ad20065fdbf\") " Oct 30 06:13:24.950290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.949951 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecabf567-9ebd-4831-ad26-6ad20065fdbf-archives-path" pod "ecabf567-9ebd-4831-ad26-6ad20065fdbf" (UID: "ecabf567-9ebd-4831-ad26-6ad20065fdbf"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:24.950290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.949973 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-service-ca-bundle" pod "ecabf567-9ebd-4831-ad26-6ad20065fdbf" (UID: "ecabf567-9ebd-4831-ad26-6ad20065fdbf"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:24.950290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.950017 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2jx\" (UniqueName: \"kubernetes.io/projected/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-kube-api-access-6c2jx\") pod \"maas-api-revoke-sub-6c4f2d58-wdt65\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:24.950290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.950188 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c4f2d58-wdt65\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:24.950633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.950354 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c4f2d58-wdt65\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:24.950706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.950690 5001 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-service-ca-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:24.950765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.950724 5001 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ecabf567-9ebd-4831-ad26-6ad20065fdbf-archives-path\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:24.950765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.950741 5001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecabf567-9ebd-4831-ad26-6ad20065fdbf-trusted-ca-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:24.951007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.950966 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:24.951056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.951042 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:24.951090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.951074 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:24.951652 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:24.951553 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecabf567-9ebd-4831-ad26-6ad20065fdbf-kube-api-access-8t287" pod "ecabf567-9ebd-4831-ad26-6ad20065fdbf" (UID: "ecabf567-9ebd-4831-ad26-6ad20065fdbf"). InnerVolumeSpecName "kube-api-access-8t287". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:25.052188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.052154 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c4f2d58-wdt65\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:25.052349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.052297 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c4f2d58-wdt65\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:25.052521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.052491 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2jx\" (UniqueName: \"kubernetes.io/projected/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-kube-api-access-6c2jx\") pod \"maas-api-revoke-sub-6c4f2d58-wdt65\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:25.052682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.052661 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t287\" (UniqueName: \"kubernetes.io/projected/ecabf567-9ebd-4831-ad26-6ad20065fdbf-kube-api-access-8t287\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:25.052948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.052925 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c4f2d58-wdt65\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:25.054476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.054453 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c4f2d58-wdt65\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:25.077224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.077189 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2jx\" (UniqueName: \"kubernetes.io/projected/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-kube-api-access-6c2jx\") pod \"maas-api-revoke-sub-6c4f2d58-wdt65\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:25.135596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.135554 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:25.443135 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:13:25.443104 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cb43cc_5dda_4675_9f59_29a42f2cdbd3.slice/crio-5c8eeaacde7e1733ef8369584dce13098dcb94fbf031e47637f96900dd0fccf2 WatchSource:0}: Error finding container 5c8eeaacde7e1733ef8369584dce13098dcb94fbf031e47637f96900dd0fccf2: Status 404 returned error can't find the container with id 5c8eeaacde7e1733ef8369584dce13098dcb94fbf031e47637f96900dd0fccf2 Oct 30 06:13:25.443868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.443813 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65"] Oct 30 06:13:25.471591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.471563 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" event={"ID":"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3","Type":"ContainerStarted","Data":"5c8eeaacde7e1733ef8369584dce13098dcb94fbf031e47637f96900dd0fccf2"} Oct 30 06:13:25.473018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.473000 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" Oct 30 06:13:25.473132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.473007 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jzt6v-mwmx8" event={"ID":"ecabf567-9ebd-4831-ad26-6ad20065fdbf","Type":"ContainerDied","Data":"4eb29e0d15ea07f2726bf92f49097cc996da3ff00033fd3d314b3422bb1e6ffd"} Oct 30 06:13:25.473132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.473054 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eb29e0d15ea07f2726bf92f49097cc996da3ff00033fd3d314b3422bb1e6ffd" Oct 30 06:13:25.474757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.474734 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" event={"ID":"f67c14bb-f110-45db-b9d8-4800a84b7277","Type":"ContainerStarted","Data":"f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52"} Oct 30 06:13:25.474837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.474780 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" event={"ID":"f67c14bb-f110-45db-b9d8-4800a84b7277","Type":"ContainerStarted","Data":"85ebe40318517ed8856608b17249d8070a43eb75c4dd403eae35a30d2ca94af4"} Oct 30 06:13:25.522603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:25.522561 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" podStartSLOduration=2.522547374 podStartE2EDuration="2.522547374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:25.519712323 +0000 UTC m=+1219.515669334" watchObservedRunningTime="2026-10-30 06:13:25.522547374 +0000 UTC m=+1219.518504384" Oct 30 06:13:26.006500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.006468 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh"] Oct 30 06:13:26.009370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.007180 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ecabf567-9ebd-4831-ad26-6ad20065fdbf" containerName="insights-gathering" Oct 30 06:13:26.009370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.007202 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecabf567-9ebd-4831-ad26-6ad20065fdbf" containerName="insights-gathering" Oct 30 06:13:26.009370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.007454 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecabf567-9ebd-4831-ad26-6ad20065fdbf" containerName="insights-gathering" Oct 30 06:13:26.010852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.010822 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.043857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.043828 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh"] Oct 30 06:13:26.077478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.077445 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzr6\" (UniqueName: \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-kube-api-access-wkzr6\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.077665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.077491 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.077712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.077656 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.077753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.077717 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.077853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.077829 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.077907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.077866 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.077907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.077899 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.078011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.077958 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.078011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.077990 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.213545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.212784 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.213545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.212844 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.213545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.212876 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.213545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.213187 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzr6\" (UniqueName: \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-kube-api-access-wkzr6\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.213545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.213230 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.213545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.213365 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.213545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.213420 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.213545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.213465 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.213545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.213489 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.234930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.234867 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.235528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.235504 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.236336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.236294 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.241610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.241584 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.241884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.241850 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.315348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.296227 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.315348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.305063 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.315348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.306653 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzr6\" (UniqueName: \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-kube-api-access-wkzr6\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.324856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.324806 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.328178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.328148 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:26.328445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.328351 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:26.336431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.336398 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:26.491102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.491018 5001 generic.go:356] "Generic (PLEG): container finished" podID="d1cb43cc-5dda-4675-9f59-29a42f2cdbd3" containerID="f03ddb2377c8dde2df34e0cd7d376224aa3ad3bb5cf1809288ce356b2f4d84d2" exitCode=0 Oct 30 06:13:26.491513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.491442 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" event={"ID":"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3","Type":"ContainerDied","Data":"f03ddb2377c8dde2df34e0cd7d376224aa3ad3bb5cf1809288ce356b2f4d84d2"} Oct 30 06:13:26.492140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.491762 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:26.493216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.493196 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:13:26.685113 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:13:26.685086 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69af5daa_6e0f_416c_b808_8a2ef5eaa603.slice/crio-281f4be85bdbe874a34d7047a37f792e170bdbe2e7cdc9a22f2eb273a3a23128 WatchSource:0}: Error finding container 281f4be85bdbe874a34d7047a37f792e170bdbe2e7cdc9a22f2eb273a3a23128: Status 404 returned error can't find the container with id 281f4be85bdbe874a34d7047a37f792e170bdbe2e7cdc9a22f2eb273a3a23128 Oct 30 06:13:26.685533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.685512 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh"] Oct 30 06:13:26.704361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.688488 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:26.704361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.688576 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:26.704361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:26.688619 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:27.508474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.508438 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" event={"ID":"69af5daa-6e0f-416c-b808-8a2ef5eaa603","Type":"ContainerStarted","Data":"f99f9c26fbb087955cb3c18faeb4002b832f2982bd1335137205e2ced33b6eb8"} Oct 30 06:13:27.509039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.508495 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" event={"ID":"69af5daa-6e0f-416c-b808-8a2ef5eaa603","Type":"ContainerStarted","Data":"281f4be85bdbe874a34d7047a37f792e170bdbe2e7cdc9a22f2eb273a3a23128"} Oct 30 06:13:27.595491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.593909 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" podStartSLOduration=2.593886446 podStartE2EDuration="2.593886446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:27.588527377 +0000 UTC m=+1221.584484388" watchObservedRunningTime="2026-10-30 06:13:27.593886446 +0000 UTC m=+1221.589843457" Oct 30 06:13:27.741092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.741068 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:27.859392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.858826 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-kube-api-access-6c2jx\" (UniqueName: \"kubernetes.io/projected/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-kube-api-access-6c2jx\") pod \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " Oct 30 06:13:27.859392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.858894 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-openshift-ca\") pod \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " Oct 30 06:13:27.859392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.858938 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-xks-ca\") pod \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\" (UID: \"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3\") " Oct 30 06:13:27.884694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.884520 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-openshift-ca" pod "d1cb43cc-5dda-4675-9f59-29a42f2cdbd3" (UID: "d1cb43cc-5dda-4675-9f59-29a42f2cdbd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:27.890021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.889989 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-xks-ca" pod "d1cb43cc-5dda-4675-9f59-29a42f2cdbd3" (UID: "d1cb43cc-5dda-4675-9f59-29a42f2cdbd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:13:27.890403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.890363 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-kube-api-access-6c2jx" pod "d1cb43cc-5dda-4675-9f59-29a42f2cdbd3" (UID: "d1cb43cc-5dda-4675-9f59-29a42f2cdbd3"). InnerVolumeSpecName "kube-api-access-6c2jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:27.981329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.980596 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c2jx\" (UniqueName: \"kubernetes.io/projected/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-kube-api-access-6c2jx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:27.981329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.980640 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:27.981329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:27.980656 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:28.344345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:28.343663 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:28.349329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:28.347440 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:13:28.349329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:28.347503 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 06:13:28.596198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:28.596118 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" Oct 30 06:13:28.609934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:28.596623 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65" event={"ID":"d1cb43cc-5dda-4675-9f59-29a42f2cdbd3","Type":"ContainerDied","Data":"5c8eeaacde7e1733ef8369584dce13098dcb94fbf031e47637f96900dd0fccf2"} Oct 30 06:13:28.610191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:28.610175 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c8eeaacde7e1733ef8369584dce13098dcb94fbf031e47637f96900dd0fccf2" Oct 30 06:13:29.329663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:29.329626 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:13:29.329844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:29.329681 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 06:13:30.033044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.033008 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx"] Oct 30 06:13:30.033933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.033911 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d1cb43cc-5dda-4675-9f59-29a42f2cdbd3" containerName="revoke-keys" Oct 30 06:13:30.034034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.033934 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cb43cc-5dda-4675-9f59-29a42f2cdbd3" containerName="revoke-keys" Oct 30 06:13:30.034142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.034128 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1cb43cc-5dda-4675-9f59-29a42f2cdbd3" containerName="revoke-keys" Oct 30 06:13:30.037052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.037031 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:13:30.061656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.061627 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e6df62-dockercfg-bfdw8\"" Oct 30 06:13:30.095381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.095343 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx"] Oct 30 06:13:30.112838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.112788 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e6df62\"" Oct 30 06:13:30.333386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.329722 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:13:30.333667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.333629 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 06:13:30.494332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.457252 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3c9884-90ba-4058-a58b-da51bb4008ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx\" (UID: \"dd3c9884-90ba-4058-a58b-da51bb4008ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:13:30.496081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.496047 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgplp\" (UniqueName: \"kubernetes.io/projected/dd3c9884-90ba-4058-a58b-da51bb4008ea-kube-api-access-sgplp\") pod \"payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx\" (UID: \"dd3c9884-90ba-4058-a58b-da51bb4008ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:13:30.599972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.599865 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgplp\" (UniqueName: \"kubernetes.io/projected/dd3c9884-90ba-4058-a58b-da51bb4008ea-kube-api-access-sgplp\") pod \"payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx\" (UID: \"dd3c9884-90ba-4058-a58b-da51bb4008ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:13:30.603096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.603064 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3c9884-90ba-4058-a58b-da51bb4008ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx\" (UID: \"dd3c9884-90ba-4058-a58b-da51bb4008ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:13:30.611388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.604238 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3c9884-90ba-4058-a58b-da51bb4008ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx\" (UID: \"dd3c9884-90ba-4058-a58b-da51bb4008ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:13:30.678682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.678649 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgplp\" (UniqueName: \"kubernetes.io/projected/dd3c9884-90ba-4058-a58b-da51bb4008ea-kube-api-access-sgplp\") pod \"payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx\" (UID: \"dd3c9884-90ba-4058-a58b-da51bb4008ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:13:30.950230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:30.950194 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:13:31.298277 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:13:31.298241 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3c9884_90ba_4058_a58b_da51bb4008ea.slice/crio-216cb7cce32f30eb1a25087b6ca2d6317019ce5f0b80127e0853f8f7e4456a79 WatchSource:0}: Error finding container 216cb7cce32f30eb1a25087b6ca2d6317019ce5f0b80127e0853f8f7e4456a79: Status 404 returned error can't find the container with id 216cb7cce32f30eb1a25087b6ca2d6317019ce5f0b80127e0853f8f7e4456a79 Oct 30 06:13:31.299388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:31.299366 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx"] Oct 30 06:13:31.331808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:31.331780 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:13:31.331944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:31.331831 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 06:13:31.690155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:31.690117 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" event={"ID":"dd3c9884-90ba-4058-a58b-da51bb4008ea","Type":"ContainerStarted","Data":"566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624"} Oct 30 06:13:31.690412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:31.690173 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" event={"ID":"dd3c9884-90ba-4058-a58b-da51bb4008ea","Type":"ContainerStarted","Data":"216cb7cce32f30eb1a25087b6ca2d6317019ce5f0b80127e0853f8f7e4456a79"} Oct 30 06:13:31.690412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:31.690221 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:13:31.734150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:31.734102 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" podStartSLOduration=1.734085995 podStartE2EDuration="1.734085995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:31.731101079 +0000 UTC m=+1225.727058090" watchObservedRunningTime="2026-10-30 06:13:31.734085995 +0000 UTC m=+1225.730043005" Oct 30 06:13:32.338336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:32.333695 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:13:32.338336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:32.333773 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 06:13:33.329576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:33.329546 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:13:33.329768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:33.329596 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 06:13:34.336336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:34.336111 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:13:34.336336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:34.336174 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 06:13:35.221331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.216215 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7"] Oct 30 06:13:35.246328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.241704 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:35.300110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.298244 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:35.300110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.298327 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:35.300110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.298503 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vfmh\" (UniqueName: \"kubernetes.io/projected/50192ebb-0573-407b-abc6-ed886723bf08-kube-api-access-4vfmh\") pod \"maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:35.302386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.302200 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7"] Oct 30 06:13:35.314703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.314543 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:13:35.315542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.315522 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:13:35.315781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.315764 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:13:35.317568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.316338 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:13:35.404019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.399882 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vfmh\" (UniqueName: \"kubernetes.io/projected/50192ebb-0573-407b-abc6-ed886723bf08-kube-api-access-4vfmh\") pod \"maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:35.404019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.400155 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:35.404019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.400205 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:35.404019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.401765 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:35.404019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.403424 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:35.426826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.426797 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vfmh\" (UniqueName: \"kubernetes.io/projected/50192ebb-0573-407b-abc6-ed886723bf08-kube-api-access-4vfmh\") pod \"maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:35.625424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:35.625396 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:36.106288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:36.104583 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7"] Oct 30 06:13:36.388832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:36.384416 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:13:36.388832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:36.384474 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:13:36.836396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:36.831401 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" event={"ID":"50192ebb-0573-407b-abc6-ed886723bf08","Type":"ContainerStarted","Data":"8c8a53770c1339dd9208d82dcbd200fed9f045ab3680e2a8b4d2a64b24a2f94e"} Oct 30 06:13:36.836396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:36.831469 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" event={"ID":"50192ebb-0573-407b-abc6-ed886723bf08","Type":"ContainerStarted","Data":"86e9b230cbb1029f8c4eed13facfc87e9b6ff608e87c8e6bae951272f083113c"} Oct 30 06:13:36.888332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:36.882289 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" podStartSLOduration=1.882271807 podStartE2EDuration="1.882271807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:36.881758868 +0000 UTC m=+1230.877715879" watchObservedRunningTime="2026-10-30 06:13:36.882271807 +0000 UTC m=+1230.878228818" Oct 30 06:13:37.446331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:37.435505 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:13:37.458341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:37.454248 5001 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:13:37.458341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:37.454329 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:13:37.843378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:37.842354 5001 generic.go:356] "Generic (PLEG): container finished" podID="50192ebb-0573-407b-abc6-ed886723bf08" containerID="8c8a53770c1339dd9208d82dcbd200fed9f045ab3680e2a8b4d2a64b24a2f94e" exitCode=0 Oct 30 06:13:37.843378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:37.842412 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" event={"ID":"50192ebb-0573-407b-abc6-ed886723bf08","Type":"ContainerDied","Data":"8c8a53770c1339dd9208d82dcbd200fed9f045ab3680e2a8b4d2a64b24a2f94e"} Oct 30 06:13:37.909352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:37.909297 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:37.910157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:37.910135 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:37.910856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:37.910836 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:13:38.296178 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:13:38.296144 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:13:39.140468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.140442 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:39.250888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.250859 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-xks-ca\") pod \"50192ebb-0573-407b-abc6-ed886723bf08\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " Oct 30 06:13:39.251161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.251149 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-openshift-ca\") pod \"50192ebb-0573-407b-abc6-ed886723bf08\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " Oct 30 06:13:39.251289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.251279 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50192ebb-0573-407b-abc6-ed886723bf08-kube-api-access-4vfmh\" (UniqueName: \"kubernetes.io/projected/50192ebb-0573-407b-abc6-ed886723bf08-kube-api-access-4vfmh\") pod \"50192ebb-0573-407b-abc6-ed886723bf08\" (UID: \"50192ebb-0573-407b-abc6-ed886723bf08\") " Oct 30 06:13:39.275332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.272509 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-openshift-ca" pod "50192ebb-0573-407b-abc6-ed886723bf08" (UID: "50192ebb-0573-407b-abc6-ed886723bf08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:39.277627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.277597 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:39.278764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.278735 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-xks-ca" pod "50192ebb-0573-407b-abc6-ed886723bf08" (UID: "50192ebb-0573-407b-abc6-ed886723bf08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:13:39.289611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.288949 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50192ebb-0573-407b-abc6-ed886723bf08-kube-api-access-4vfmh" pod "50192ebb-0573-407b-abc6-ed886723bf08" (UID: "50192ebb-0573-407b-abc6-ed886723bf08"). InnerVolumeSpecName "kube-api-access-4vfmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:39.381331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.380829 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vfmh\" (UniqueName: \"kubernetes.io/projected/50192ebb-0573-407b-abc6-ed886723bf08-kube-api-access-4vfmh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:39.381331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.380869 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50192ebb-0573-407b-abc6-ed886723bf08-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:39.887487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.887450 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" Oct 30 06:13:39.887686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.887457 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7" event={"ID":"50192ebb-0573-407b-abc6-ed886723bf08","Type":"ContainerDied","Data":"86e9b230cbb1029f8c4eed13facfc87e9b6ff608e87c8e6bae951272f083113c"} Oct 30 06:13:39.887686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:39.887678 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e9b230cbb1029f8c4eed13facfc87e9b6ff608e87c8e6bae951272f083113c" Oct 30 06:13:42.704243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:42.704210 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:13:46.342787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:46.342694 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl"] Oct 30 06:13:47.108330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:47.102655 5001 generic.go:356] "Generic (PLEG): container finished" podID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerID="e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a" exitCode=0 Oct 30 06:13:47.108330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:47.102802 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" event={"ID":"f43aeac6-adbb-433c-92b2-3d92cb5c9660","Type":"ContainerDied","Data":"e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a"} Oct 30 06:13:47.108330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:47.103263 5001 scope.go:122] "RemoveContainer" containerID="e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a" Oct 30 06:13:47.151494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:47.126204 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:47.151494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:47.126327 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:47.151494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:47.126378 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:47.177331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:47.156053 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq"] Oct 30 06:13:47.301503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:47.301433 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b44eb62-lvcwq"] Oct 30 06:13:48.119120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:48.119089 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d88a5f-de29-4751-bfb9-947f381498c4" path="/var/lib/kubelet/pods/39d88a5f-de29-4751-bfb9-947f381498c4/volumes" Oct 30 06:13:48.119618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:48.119460 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" event={"ID":"f43aeac6-adbb-433c-92b2-3d92cb5c9660","Type":"ContainerStarted","Data":"553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b"} Oct 30 06:13:48.880331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:48.880269 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:13:48.880556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:48.880516 5001 patch_prober.go:29] interesting pod/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:13:48.880626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:48.880572 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 06:13:49.878904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:49.878861 5001 patch_prober.go:29] interesting pod/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:13:49.879441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:49.878943 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 06:13:50.345934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.345845 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4"] Oct 30 06:13:50.346597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.346579 5001 cpu_manager.go:414] "containerMap: removing container" podUID="50192ebb-0573-407b-abc6-ed886723bf08" containerName="revoke-keys" Oct 30 06:13:50.346597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.346597 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50192ebb-0573-407b-abc6-ed886723bf08" containerName="revoke-keys" Oct 30 06:13:50.346749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.346735 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="50192ebb-0573-407b-abc6-ed886723bf08" containerName="revoke-keys" Oct 30 06:13:50.350396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.350374 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:50.382332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.382150 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:13:50.382332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.382299 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:13:50.382620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.382577 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:13:50.382694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.382680 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:13:50.386908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.386877 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4"] Oct 30 06:13:50.535136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.535105 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:50.535341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.535274 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:50.535389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.535340 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w22g\" (UniqueName: \"kubernetes.io/projected/7dbf896a-d653-4437-b7d5-fb4ea05712e6-kube-api-access-4w22g\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:50.637293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.637256 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:50.637534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.637342 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w22g\" (UniqueName: \"kubernetes.io/projected/7dbf896a-d653-4437-b7d5-fb4ea05712e6-kube-api-access-4w22g\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:50.637608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.637532 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:50.638152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.638128 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:50.640000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.639978 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:50.664330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.663387 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w22g\" (UniqueName: \"kubernetes.io/projected/7dbf896a-d653-4437-b7d5-fb4ea05712e6-kube-api-access-4w22g\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:50.699746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.699709 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:50.882329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.881418 5001 patch_prober.go:29] interesting pod/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:13:50.882329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:50.881477 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 06:13:51.044894 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:13:51.044860 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbf896a_d653_4437_b7d5_fb4ea05712e6.slice/crio-39f848f1d7c1ee27277ffa96641b9274426f880a84963013f647cc78324215cc WatchSource:0}: Error finding container 39f848f1d7c1ee27277ffa96641b9274426f880a84963013f647cc78324215cc: Status 404 returned error can't find the container with id 39f848f1d7c1ee27277ffa96641b9274426f880a84963013f647cc78324215cc Oct 30 06:13:51.046612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:51.046589 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4"] Oct 30 06:13:51.143914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:51.143876 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" event={"ID":"7dbf896a-d653-4437-b7d5-fb4ea05712e6","Type":"ContainerStarted","Data":"39f848f1d7c1ee27277ffa96641b9274426f880a84963013f647cc78324215cc"} Oct 30 06:13:51.343861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:51.343771 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" podUID="3e720b79-23aa-4ff6-905e-fa74f201864d" containerName="payload-processing" containerID="cri-o://6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce" gracePeriod=25 Oct 30 06:13:51.882167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:51.881698 5001 patch_prober.go:29] interesting pod/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:13:51.882167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:51.881760 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 06:13:51.897254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:51.897234 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:13:51.989827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:51.989726 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e720b79-23aa-4ff6-905e-fa74f201864d-kube-api-access-qz69c\" (UniqueName: \"kubernetes.io/projected/3e720b79-23aa-4ff6-905e-fa74f201864d-kube-api-access-qz69c\") pod \"3e720b79-23aa-4ff6-905e-fa74f201864d\" (UID: \"3e720b79-23aa-4ff6-905e-fa74f201864d\") " Oct 30 06:13:51.990007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:51.989903 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e720b79-23aa-4ff6-905e-fa74f201864d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e720b79-23aa-4ff6-905e-fa74f201864d-plugins-config-volume\") pod \"3e720b79-23aa-4ff6-905e-fa74f201864d\" (UID: \"3e720b79-23aa-4ff6-905e-fa74f201864d\") " Oct 30 06:13:51.990570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:51.990537 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e720b79-23aa-4ff6-905e-fa74f201864d-plugins-config-volume" pod "3e720b79-23aa-4ff6-905e-fa74f201864d" (UID: "3e720b79-23aa-4ff6-905e-fa74f201864d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:51.999404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:51.999372 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e720b79-23aa-4ff6-905e-fa74f201864d-kube-api-access-qz69c" pod "3e720b79-23aa-4ff6-905e-fa74f201864d" (UID: "3e720b79-23aa-4ff6-905e-fa74f201864d"). InnerVolumeSpecName "kube-api-access-qz69c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:52.091855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.091826 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e720b79-23aa-4ff6-905e-fa74f201864d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:52.091855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.091852 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz69c\" (UniqueName: \"kubernetes.io/projected/3e720b79-23aa-4ff6-905e-fa74f201864d-kube-api-access-qz69c\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:52.182951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.182282 5001 generic.go:356] "Generic (PLEG): container finished" podID="3e720b79-23aa-4ff6-905e-fa74f201864d" containerID="6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce" exitCode=0 Oct 30 06:13:52.182951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.182461 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" Oct 30 06:13:52.184422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.183791 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" event={"ID":"3e720b79-23aa-4ff6-905e-fa74f201864d","Type":"ContainerDied","Data":"6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce"} Oct 30 06:13:52.184422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.183848 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl" event={"ID":"3e720b79-23aa-4ff6-905e-fa74f201864d","Type":"ContainerDied","Data":"8e7f19d3842346ecdd353aacd4636d8e9b3e2aa51a2e4a833434c83b1a09e6be"} Oct 30 06:13:52.184422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.183879 5001 scope.go:122] "RemoveContainer" containerID="6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce" Oct 30 06:13:52.188348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.187751 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" event={"ID":"7dbf896a-d653-4437-b7d5-fb4ea05712e6","Type":"ContainerStarted","Data":"83b0b94a7f62bf7a2f544a30b1f46e03c84d3043ea6d1e56c8d72aed3edbb720"} Oct 30 06:13:52.249160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.249077 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" podStartSLOduration=2.249063946 podStartE2EDuration="2.249063946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:52.244795116 +0000 UTC m=+1246.240752129" watchObservedRunningTime="2026-10-30 06:13:52.249063946 +0000 UTC m=+1246.245020946" Oct 30 06:13:52.251295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.251274 5001 scope.go:122] "RemoveContainer" containerID="6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce" Oct 30 06:13:52.257327 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:13:52.256060 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce\": container with ID starting with 6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce not found: ID does not exist" containerID="6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce" Oct 30 06:13:52.257327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.256101 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce"} err="failed to get container status \"6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce\": rpc error: code = NotFound desc = could not find container \"6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce\": container with ID starting with 6808b1419eee49995eab13f5f1065a16be159353a6e37b6b48d1fc192bd45dce not found: ID does not exist" Oct 30 06:13:52.302827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.301813 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl"] Oct 30 06:13:52.310697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.310663 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ec766-77bd9f9458-ls5rl"] Oct 30 06:13:52.903327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.893704 5001 patch_prober.go:29] interesting pod/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:13:52.903327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:52.893785 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 06:13:53.201464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:53.201374 5001 generic.go:356] "Generic (PLEG): container finished" podID="7dbf896a-d653-4437-b7d5-fb4ea05712e6" containerID="83b0b94a7f62bf7a2f544a30b1f46e03c84d3043ea6d1e56c8d72aed3edbb720" exitCode=0 Oct 30 06:13:53.201464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:53.201424 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" event={"ID":"7dbf896a-d653-4437-b7d5-fb4ea05712e6","Type":"ContainerDied","Data":"83b0b94a7f62bf7a2f544a30b1f46e03c84d3043ea6d1e56c8d72aed3edbb720"} Oct 30 06:13:53.884591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:53.884556 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:13:53.884968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:53.884939 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:13:53.885707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:53.885688 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:13:54.124653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:54.124624 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e720b79-23aa-4ff6-905e-fa74f201864d" path="/var/lib/kubelet/pods/3e720b79-23aa-4ff6-905e-fa74f201864d/volumes" Oct 30 06:13:54.385035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:54.384998 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:54.456991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:54.456959 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-openshift-ca\") pod \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " Oct 30 06:13:54.457182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:54.457130 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-xks-ca\") pod \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " Oct 30 06:13:54.457182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:54.457168 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dbf896a-d653-4437-b7d5-fb4ea05712e6-kube-api-access-4w22g\" (UniqueName: \"kubernetes.io/projected/7dbf896a-d653-4437-b7d5-fb4ea05712e6-kube-api-access-4w22g\") pod \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\" (UID: \"7dbf896a-d653-4437-b7d5-fb4ea05712e6\") " Oct 30 06:13:54.457420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:54.457392 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-openshift-ca" pod "7dbf896a-d653-4437-b7d5-fb4ea05712e6" (UID: "7dbf896a-d653-4437-b7d5-fb4ea05712e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:54.457784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:54.457760 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:54.459493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:54.459465 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-xks-ca" pod "7dbf896a-d653-4437-b7d5-fb4ea05712e6" (UID: "7dbf896a-d653-4437-b7d5-fb4ea05712e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:13:54.460258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:54.460205 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbf896a-d653-4437-b7d5-fb4ea05712e6-kube-api-access-4w22g" pod "7dbf896a-d653-4437-b7d5-fb4ea05712e6" (UID: "7dbf896a-d653-4437-b7d5-fb4ea05712e6"). InnerVolumeSpecName "kube-api-access-4w22g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:54.559357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:54.559321 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbf896a-d653-4437-b7d5-fb4ea05712e6-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:54.559357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:54.559355 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w22g\" (UniqueName: \"kubernetes.io/projected/7dbf896a-d653-4437-b7d5-fb4ea05712e6-kube-api-access-4w22g\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:13:55.220216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:55.220192 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" Oct 30 06:13:55.220665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:55.220231 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4" event={"ID":"7dbf896a-d653-4437-b7d5-fb4ea05712e6","Type":"ContainerDied","Data":"39f848f1d7c1ee27277ffa96641b9274426f880a84963013f647cc78324215cc"} Oct 30 06:13:55.220665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:55.220286 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39f848f1d7c1ee27277ffa96641b9274426f880a84963013f647cc78324215cc" Oct 30 06:13:57.917905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:57.917862 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:13:58.087495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:58.086281 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh"] Oct 30 06:13:58.091879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:58.091844 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3b890e-tzzgh"] Oct 30 06:13:58.126072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:13:58.126041 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afe7122-ca01-4c70-87c9-b03e115d8575" path="/var/lib/kubelet/pods/3afe7122-ca01-4c70-87c9-b03e115d8575/volumes" Oct 30 06:14:00.983327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:00.982679 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh"] Oct 30 06:14:01.009393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:01.009357 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9"] Oct 30 06:14:01.094076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:01.094044 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz"] Oct 30 06:14:01.106389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:01.106357 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5d795c-t8kdz"] Oct 30 06:14:02.105331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:02.100369 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg"] Oct 30 06:14:02.217330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:02.196655 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269b438f-e946-4b57-b782-d428128c2c9a" path="/var/lib/kubelet/pods/269b438f-e946-4b57-b782-d428128c2c9a/volumes" Oct 30 06:14:02.217330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:02.197138 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46dbef1c-rsbqg"] Oct 30 06:14:02.756625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:02.755401 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:14:03.280683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:03.276730 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:14:04.119006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:04.118975 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54" path="/var/lib/kubelet/pods/b60f6d92-1eef-463f-a9a3-2fcdd6d0cb54/volumes" Oct 30 06:14:05.983812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:05.983707 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" podUID="142a47dc-0a88-4e03-b0b7-8a62d8b8e861" containerName="payload-processing" containerID="cri-o://3c2b21d10b2791e311806d6a3c006ef85c358fb50a9b94adf80aaec1c7d33e7f" gracePeriod=25 Oct 30 06:14:06.010696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:06.010657 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" podUID="dc51944b-1255-4681-a6dc-fc6e503116b5" containerName="payload-pre-processing" containerID="cri-o://38c120a9d7ea02f0a2c3d955a262bb04ea8fccafa61502284760c8520ca5f798" gracePeriod=25 Oct 30 06:14:08.250287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.250254 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv"] Oct 30 06:14:08.250950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.250935 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7dbf896a-d653-4437-b7d5-fb4ea05712e6" containerName="revoke-keys" Oct 30 06:14:08.251013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.250953 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dbf896a-d653-4437-b7d5-fb4ea05712e6" containerName="revoke-keys" Oct 30 06:14:08.251013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.250988 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3e720b79-23aa-4ff6-905e-fa74f201864d" containerName="payload-processing" Oct 30 06:14:08.251013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.250994 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e720b79-23aa-4ff6-905e-fa74f201864d" containerName="payload-processing" Oct 30 06:14:08.251120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.251095 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e720b79-23aa-4ff6-905e-fa74f201864d" containerName="payload-processing" Oct 30 06:14:08.251120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.251110 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dbf896a-d653-4437-b7d5-fb4ea05712e6" containerName="revoke-keys" Oct 30 06:14:08.274609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.274580 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv"] Oct 30 06:14:08.274776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.274674 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:08.277037 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:14:08.277004 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:14:08.290372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.290251 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:14:08.290563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.290505 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:14:08.290736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.290713 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:14:08.303443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.303412 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:14:08.442242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.442207 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtfw\" (UniqueName: \"kubernetes.io/projected/89e3754b-41d7-468f-bd6a-153bd16045a8-kube-api-access-pmtfw\") pod \"maas-api-revoke-sub-c3267419-tmkrv\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:08.442447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.442264 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3267419-tmkrv\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:08.442536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.442519 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3267419-tmkrv\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:08.554757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.554659 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3267419-tmkrv\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:08.554950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.554884 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtfw\" (UniqueName: \"kubernetes.io/projected/89e3754b-41d7-468f-bd6a-153bd16045a8-kube-api-access-pmtfw\") pod \"maas-api-revoke-sub-c3267419-tmkrv\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:08.554950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.554933 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3267419-tmkrv\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:08.557495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.555937 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3267419-tmkrv\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:08.568151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.568118 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3267419-tmkrv\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:08.580521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.580495 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtfw\" (UniqueName: \"kubernetes.io/projected/89e3754b-41d7-468f-bd6a-153bd16045a8-kube-api-access-pmtfw\") pod \"maas-api-revoke-sub-c3267419-tmkrv\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:08.598138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.598114 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:08.931746 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:14:08.931710 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e3754b_41d7_468f_bd6a_153bd16045a8.slice/crio-2e9e9413f2b3cd9653c533cf738daed49df3a7a5db5b8a5cba6aeda52dd4fd4b WatchSource:0}: Error finding container 2e9e9413f2b3cd9653c533cf738daed49df3a7a5db5b8a5cba6aeda52dd4fd4b: Status 404 returned error can't find the container with id 2e9e9413f2b3cd9653c533cf738daed49df3a7a5db5b8a5cba6aeda52dd4fd4b Oct 30 06:14:08.934937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:08.934906 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv"] Oct 30 06:14:09.261240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.261162 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" podUID="142a47dc-0a88-4e03-b0b7-8a62d8b8e861" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:14:09.801032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.801000 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b"] Oct 30 06:14:09.804697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.804671 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:09.827905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.824105 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-fe108e-openshift-default-dockercfg-82tsd\"" Oct 30 06:14:09.845218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.842760 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b"] Oct 30 06:14:09.883165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.883137 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-token\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:09.883432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.883411 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-socket\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:09.883520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.883450 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-podinfo\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:09.883585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.883523 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cfbab7a-23c7-4712-b0c6-65529f21890e-istiod-ca-cert\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:09.883645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.883613 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-credential-socket\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:09.883811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.883790 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jl4b\" (UniqueName: \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-kube-api-access-9jl4b\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:09.883886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.883824 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-certs\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:09.883886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.883842 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-envoy\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:09.883886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.883856 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-data\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:09.890015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.889992 5001 generic.go:356] "Generic (PLEG): container finished" podID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerID="f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52" exitCode=0 Oct 30 06:14:09.890123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.890069 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" event={"ID":"f67c14bb-f110-45db-b9d8-4800a84b7277","Type":"ContainerDied","Data":"f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52"} Oct 30 06:14:09.890568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.890550 5001 scope.go:122] "RemoveContainer" containerID="f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52" Oct 30 06:14:09.892040 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.892013 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:09.892119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.892068 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" event={"ID":"89e3754b-41d7-468f-bd6a-153bd16045a8","Type":"ContainerStarted","Data":"9ba901a8870bb6b63fe5840a9431d5d417034c55fccdb4f34d9b73ee76d5402c"} Oct 30 06:14:09.892159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.892119 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" event={"ID":"89e3754b-41d7-468f-bd6a-153bd16045a8","Type":"ContainerStarted","Data":"2e9e9413f2b3cd9653c533cf738daed49df3a7a5db5b8a5cba6aeda52dd4fd4b"} Oct 30 06:14:09.892159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.892078 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:09.892246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.892225 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.990491 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-socket\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.990538 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-podinfo\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.990619 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cfbab7a-23c7-4712-b0c6-65529f21890e-istiod-ca-cert\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.990680 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-credential-socket\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.990810 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jl4b\" (UniqueName: \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-kube-api-access-9jl4b\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.990839 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-certs\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.990865 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-envoy\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.990886 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-data\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.991162 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-token\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:09.991855 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-socket\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.011594 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cfbab7a-23c7-4712-b0c6-65529f21890e-istiod-ca-cert\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.011994 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-credential-socket\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.012770 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-data\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.026683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.013058 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-certs\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.193457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.069154 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-podinfo\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.312026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.311884 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-envoy\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.312522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.312183 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jl4b\" (UniqueName: \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-kube-api-access-9jl4b\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.316277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.314441 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" podUID="dc51944b-1255-4681-a6dc-fc6e503116b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:14:10.338707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.338582 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-token\") pod \"e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.348716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.348668 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" podStartSLOduration=2.34865353 podStartE2EDuration="2.34865353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:10.028408702 +0000 UTC m=+1264.024365761" watchObservedRunningTime="2026-10-30 06:14:10.34865353 +0000 UTC m=+1264.344610541" Oct 30 06:14:10.468343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.456007 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:10.923753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.923719 5001 generic.go:356] "Generic (PLEG): container finished" podID="89e3754b-41d7-468f-bd6a-153bd16045a8" containerID="9ba901a8870bb6b63fe5840a9431d5d417034c55fccdb4f34d9b73ee76d5402c" exitCode=0 Oct 30 06:14:10.924206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.924179 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" event={"ID":"89e3754b-41d7-468f-bd6a-153bd16045a8","Type":"ContainerDied","Data":"9ba901a8870bb6b63fe5840a9431d5d417034c55fccdb4f34d9b73ee76d5402c"} Oct 30 06:14:10.928376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.928299 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" event={"ID":"f67c14bb-f110-45db-b9d8-4800a84b7277","Type":"ContainerStarted","Data":"cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d"} Oct 30 06:14:10.942353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.942331 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b"] Oct 30 06:14:10.944144 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:14:10.944119 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfbab7a_23c7_4712_b0c6_65529f21890e.slice/crio-9ee60c8fbca4ae1ea567334aa4e68521fb7d246f1f08122d6259a0f968634be6 WatchSource:0}: Error finding container 9ee60c8fbca4ae1ea567334aa4e68521fb7d246f1f08122d6259a0f968634be6: Status 404 returned error can't find the container with id 9ee60c8fbca4ae1ea567334aa4e68521fb7d246f1f08122d6259a0f968634be6 Oct 30 06:14:10.946780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.946750 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:10.946839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.946812 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:10.946888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:10.946845 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:11.328464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:11.328369 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:14:11.329642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:11.329618 5001 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:14:11.329760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:11.329662 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 06:14:11.492101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:11.492068 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:14:11.938401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:11.938365 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" event={"ID":"8cfbab7a-23c7-4712-b0c6-65529f21890e","Type":"ContainerStarted","Data":"1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253"} Oct 30 06:14:11.938622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:11.938425 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" event={"ID":"8cfbab7a-23c7-4712-b0c6-65529f21890e","Type":"ContainerStarted","Data":"9ee60c8fbca4ae1ea567334aa4e68521fb7d246f1f08122d6259a0f968634be6"} Oct 30 06:14:12.009720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.009671 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" podStartSLOduration=3.009656923 podStartE2EDuration="3.009656923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:12.006500559 +0000 UTC m=+1266.002457569" watchObservedRunningTime="2026-10-30 06:14:12.009656923 +0000 UTC m=+1266.005613934" Oct 30 06:14:12.347491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.347298 5001 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:14:12.348167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.348129 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 06:14:12.434233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.434210 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:12.467331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.461014 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:12.474334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.473359 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:12.551239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.549784 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e3754b-41d7-468f-bd6a-153bd16045a8-kube-api-access-pmtfw\" (UniqueName: \"kubernetes.io/projected/89e3754b-41d7-468f-bd6a-153bd16045a8-kube-api-access-pmtfw\") pod \"89e3754b-41d7-468f-bd6a-153bd16045a8\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " Oct 30 06:14:12.551239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.550133 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-openshift-ca\") pod \"89e3754b-41d7-468f-bd6a-153bd16045a8\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " Oct 30 06:14:12.551239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.550225 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-xks-ca\") pod \"89e3754b-41d7-468f-bd6a-153bd16045a8\" (UID: \"89e3754b-41d7-468f-bd6a-153bd16045a8\") " Oct 30 06:14:12.551239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.550820 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-openshift-ca" pod "89e3754b-41d7-468f-bd6a-153bd16045a8" (UID: "89e3754b-41d7-468f-bd6a-153bd16045a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:12.594083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.594037 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-xks-ca" pod "89e3754b-41d7-468f-bd6a-153bd16045a8" (UID: "89e3754b-41d7-468f-bd6a-153bd16045a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:12.594537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.594504 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e3754b-41d7-468f-bd6a-153bd16045a8-kube-api-access-pmtfw" pod "89e3754b-41d7-468f-bd6a-153bd16045a8" (UID: "89e3754b-41d7-468f-bd6a-153bd16045a8"). InnerVolumeSpecName "kube-api-access-pmtfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:12.665333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.652243 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:12.665333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.652281 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e3754b-41d7-468f-bd6a-153bd16045a8-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:12.665333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.652297 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmtfw\" (UniqueName: \"kubernetes.io/projected/89e3754b-41d7-468f-bd6a-153bd16045a8-kube-api-access-pmtfw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:12.948108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.948081 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" Oct 30 06:14:12.948289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.948107 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv" event={"ID":"89e3754b-41d7-468f-bd6a-153bd16045a8","Type":"ContainerDied","Data":"2e9e9413f2b3cd9653c533cf738daed49df3a7a5db5b8a5cba6aeda52dd4fd4b"} Oct 30 06:14:12.948289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.948153 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e9e9413f2b3cd9653c533cf738daed49df3a7a5db5b8a5cba6aeda52dd4fd4b" Oct 30 06:14:12.948695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.948661 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:12.949414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:12.949398 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:14:13.328645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:13.328552 5001 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:14:13.328645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:13.328609 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 06:14:14.328044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:14.328005 5001 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:14:14.328558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:14.328063 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 06:14:15.328222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:15.328189 5001 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:14:15.328593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:15.328232 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 06:14:16.328261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.328226 5001 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:14:16.328735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.328279 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 06:14:16.426496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.426458 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s"] Oct 30 06:14:16.427200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.427179 5001 cpu_manager.go:414] "containerMap: removing container" podUID="89e3754b-41d7-468f-bd6a-153bd16045a8" containerName="revoke-keys" Oct 30 06:14:16.427329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.427202 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e3754b-41d7-468f-bd6a-153bd16045a8" containerName="revoke-keys" Oct 30 06:14:16.427452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.427438 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e3754b-41d7-468f-bd6a-153bd16045a8" containerName="revoke-keys" Oct 30 06:14:16.430268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.430248 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:14:16.450411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.450384 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s"] Oct 30 06:14:16.465345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.457347 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-fe108e-dockercfg-hl82n\"" Oct 30 06:14:16.465345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.457493 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-fe108e\"" Oct 30 06:14:16.573329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.573207 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da8e900-94fb-4359-b1d9-25dd84b75281-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s\" (UID: \"8da8e900-94fb-4359-b1d9-25dd84b75281\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:14:16.589390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.588271 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kc5\" (UniqueName: \"kubernetes.io/projected/8da8e900-94fb-4359-b1d9-25dd84b75281-kube-api-access-d9kc5\") pod \"payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s\" (UID: \"8da8e900-94fb-4359-b1d9-25dd84b75281\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:14:16.620331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.620164 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff"] Oct 30 06:14:16.677330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.670347 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:14:16.694332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.691186 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da8e900-94fb-4359-b1d9-25dd84b75281-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s\" (UID: \"8da8e900-94fb-4359-b1d9-25dd84b75281\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:14:16.694332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.691424 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kc5\" (UniqueName: \"kubernetes.io/projected/8da8e900-94fb-4359-b1d9-25dd84b75281-kube-api-access-d9kc5\") pod \"payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s\" (UID: \"8da8e900-94fb-4359-b1d9-25dd84b75281\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:14:16.694332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.692854 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da8e900-94fb-4359-b1d9-25dd84b75281-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s\" (UID: \"8da8e900-94fb-4359-b1d9-25dd84b75281\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:14:16.754356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.751796 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kc5\" (UniqueName: \"kubernetes.io/projected/8da8e900-94fb-4359-b1d9-25dd84b75281-kube-api-access-d9kc5\") pod \"payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s\" (UID: \"8da8e900-94fb-4359-b1d9-25dd84b75281\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:14:16.800795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.800369 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff"] Oct 30 06:14:16.807526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.807073 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff\" (UID: \"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:14:16.807526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:16.807327 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q69xf\" (UniqueName: \"kubernetes.io/projected/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-kube-api-access-q69xf\") pod \"payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff\" (UID: \"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:14:17.096245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.095364 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:14:17.135198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.135157 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff\" (UID: \"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:14:17.135462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.135334 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q69xf\" (UniqueName: \"kubernetes.io/projected/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-kube-api-access-q69xf\") pod \"payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff\" (UID: \"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:14:17.139051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.138966 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff\" (UID: \"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:14:17.147688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.147603 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd"] Oct 30 06:14:17.153404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.153376 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-505032de-297pd"] Oct 30 06:14:17.176283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.176087 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q69xf\" (UniqueName: \"kubernetes.io/projected/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-kube-api-access-q69xf\") pod \"payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff\" (UID: \"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:14:17.332820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.332790 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:14:17.334060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.334038 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:14:17.376824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.376798 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:14:17.479864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.479835 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s"] Oct 30 06:14:17.698592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:17.695725 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff"] Oct 30 06:14:18.119349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.119316 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b83e190-4707-4fb6-b93e-f8304c6edf9c" path="/var/lib/kubelet/pods/1b83e190-4707-4fb6-b93e-f8304c6edf9c/volumes" Oct 30 06:14:18.153858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.153826 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" event={"ID":"8da8e900-94fb-4359-b1d9-25dd84b75281","Type":"ContainerStarted","Data":"05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc"} Oct 30 06:14:18.154047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.153886 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" event={"ID":"8da8e900-94fb-4359-b1d9-25dd84b75281","Type":"ContainerStarted","Data":"a1a940b262c78257b6330226062c5082639134c29e3b8ba04619fe885ec33a92"} Oct 30 06:14:18.154047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.153932 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:14:18.155833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.155802 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" event={"ID":"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0","Type":"ContainerStarted","Data":"92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72"} Oct 30 06:14:18.155994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.155851 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" event={"ID":"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0","Type":"ContainerStarted","Data":"57c6365476b097dffbbd205b9d76948123ac81f1d597c49bfc3449fbec689619"} Oct 30 06:14:18.155994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.155897 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:14:18.201076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.201029 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" podStartSLOduration=2.201016875 podStartE2EDuration="2.201016875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:18.195484008 +0000 UTC m=+1272.191441020" watchObservedRunningTime="2026-10-30 06:14:18.201016875 +0000 UTC m=+1272.196973884" Oct 30 06:14:18.237032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.236978 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" podStartSLOduration=2.2369579 podStartE2EDuration="2.2369579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:18.236912967 +0000 UTC m=+1272.232869982" watchObservedRunningTime="2026-10-30 06:14:18.2369579 +0000 UTC m=+1272.232914910" Oct 30 06:14:18.730933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.730904 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx"] Oct 30 06:14:18.748367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.748281 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:18.753463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.751506 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx"] Oct 30 06:14:18.765516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.765489 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:14:18.781773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.781744 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:14:18.782047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.782031 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:14:18.796474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.796441 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:14:18.925330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.908605 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd4959ac-4tcpx\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:18.925819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.925783 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv8qh\" (UniqueName: \"kubernetes.io/projected/95ad37e2-90ac-4105-8175-4245b01a3bf7-kube-api-access-pv8qh\") pod \"maas-api-revoke-sub-cd4959ac-4tcpx\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:18.926008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:18.925990 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd4959ac-4tcpx\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:19.034690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.034585 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv8qh\" (UniqueName: \"kubernetes.io/projected/95ad37e2-90ac-4105-8175-4245b01a3bf7-kube-api-access-pv8qh\") pod \"maas-api-revoke-sub-cd4959ac-4tcpx\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:19.057929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.057884 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd4959ac-4tcpx\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:19.058237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.035013 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd4959ac-4tcpx\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:19.059248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.059202 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd4959ac-4tcpx\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:19.086767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.086714 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd4959ac-4tcpx\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:19.090973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.090945 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv8qh\" (UniqueName: \"kubernetes.io/projected/95ad37e2-90ac-4105-8175-4245b01a3bf7-kube-api-access-pv8qh\") pod \"maas-api-revoke-sub-cd4959ac-4tcpx\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:19.093354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.093327 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7"] Oct 30 06:14:19.098342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.098318 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ec766-l8bh7"] Oct 30 06:14:19.258888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.256813 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" podUID="142a47dc-0a88-4e03-b0b7-8a62d8b8e861" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:14:19.369213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.369180 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:19.498405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.498372 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn"] Oct 30 06:14:19.510877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.508246 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.530538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.530511 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c15d3a-openshift-default-dockercfg-lb525\"" Oct 30 06:14:19.549472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.549441 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn"] Oct 30 06:14:19.716837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.705250 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-credential-socket\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.716837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.705375 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-podinfo\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.716837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.705431 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxhc\" (UniqueName: \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-kube-api-access-crxhc\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.716837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.705491 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-token\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.716837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.705522 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-certs\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.716837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.705712 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-socket\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.716837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.705744 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b399b8-2225-4498-aa8f-e5ea451f7131-istiod-ca-cert\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.716837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.705774 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-data\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.716837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.705801 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-envoy\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.731721 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:14:19.731683 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ad37e2_90ac_4105_8175_4245b01a3bf7.slice/crio-98cff540bc03a67b13df4210c6fd13fe022dfc20b9f4af3ffc576eb72c4b7f93 WatchSource:0}: Error finding container 98cff540bc03a67b13df4210c6fd13fe022dfc20b9f4af3ffc576eb72c4b7f93: Status 404 returned error can't find the container with id 98cff540bc03a67b13df4210c6fd13fe022dfc20b9f4af3ffc576eb72c4b7f93 Oct 30 06:14:19.760332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.759782 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx"] Oct 30 06:14:19.808695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.807289 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxhc\" (UniqueName: \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-kube-api-access-crxhc\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.808695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.807398 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-token\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.808695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.807435 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-certs\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.808695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.807640 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-socket\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.808695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.807672 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b399b8-2225-4498-aa8f-e5ea451f7131-istiod-ca-cert\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.808695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.807698 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-data\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.808695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.807726 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-envoy\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.808695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.807798 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-credential-socket\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.808695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.807920 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-podinfo\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.810168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.810132 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-socket\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.811107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.811081 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-data\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.815350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.811257 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-certs\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.828329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.816217 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b399b8-2225-4498-aa8f-e5ea451f7131-istiod-ca-cert\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.858422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.823760 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-credential-socket\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.873286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.873251 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-podinfo\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.887330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.881031 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-envoy\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.887608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.883351 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxhc\" (UniqueName: \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-kube-api-access-crxhc\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:19.900329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:19.889229 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-token\") pod \"e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:20.119707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:20.119676 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50192ebb-0573-407b-abc6-ed886723bf08" path="/var/lib/kubelet/pods/50192ebb-0573-407b-abc6-ed886723bf08/volumes" Oct 30 06:14:20.133151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:20.133114 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:20.204822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:20.204432 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" event={"ID":"95ad37e2-90ac-4105-8175-4245b01a3bf7","Type":"ContainerStarted","Data":"98cff540bc03a67b13df4210c6fd13fe022dfc20b9f4af3ffc576eb72c4b7f93"} Oct 30 06:14:20.272866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:20.272826 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" podUID="dc51944b-1255-4681-a6dc-fc6e503116b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:14:20.472013 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:14:20.471980 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b399b8_2225_4498_aa8f_e5ea451f7131.slice/crio-96a4b14a7d9d97a510050f694037f281d0632ae8e9364828ae12a01f01844f87 WatchSource:0}: Error finding container 96a4b14a7d9d97a510050f694037f281d0632ae8e9364828ae12a01f01844f87: Status 404 returned error can't find the container with id 96a4b14a7d9d97a510050f694037f281d0632ae8e9364828ae12a01f01844f87 Oct 30 06:14:20.475745 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:20.475705 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:20.475853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:20.475803 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:20.475853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:20.475846 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:20.476670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:20.476646 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn"] Oct 30 06:14:21.232582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:21.232543 5001 generic.go:356] "Generic (PLEG): container finished" podID="95ad37e2-90ac-4105-8175-4245b01a3bf7" containerID="524ec6ab7df0bec5d66f81bbd6d15e058293e68956dee798548fb0e4dc0353f7" exitCode=0 Oct 30 06:14:21.232582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:21.232577 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" event={"ID":"95ad37e2-90ac-4105-8175-4245b01a3bf7","Type":"ContainerDied","Data":"524ec6ab7df0bec5d66f81bbd6d15e058293e68956dee798548fb0e4dc0353f7"} Oct 30 06:14:21.234454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:21.234431 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" event={"ID":"24b399b8-2225-4498-aa8f-e5ea451f7131","Type":"ContainerStarted","Data":"e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0"} Oct 30 06:14:21.234568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:21.234466 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" event={"ID":"24b399b8-2225-4498-aa8f-e5ea451f7131","Type":"ContainerStarted","Data":"96a4b14a7d9d97a510050f694037f281d0632ae8e9364828ae12a01f01844f87"} Oct 30 06:14:21.322041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:21.321932 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" podStartSLOduration=2.321918717 podStartE2EDuration="2.321918717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:21.319828965 +0000 UTC m=+1275.315785979" watchObservedRunningTime="2026-10-30 06:14:21.321918717 +0000 UTC m=+1275.317875802" Oct 30 06:14:22.134054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.134024 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:22.140620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.140590 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:22.242625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.242590 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:22.243631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.243612 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:14:22.743657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.743622 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:22.850424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.849855 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-xks-ca\") pod \"95ad37e2-90ac-4105-8175-4245b01a3bf7\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " Oct 30 06:14:22.850909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.850847 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-openshift-ca" pod "95ad37e2-90ac-4105-8175-4245b01a3bf7" (UID: "95ad37e2-90ac-4105-8175-4245b01a3bf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:22.851044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.850916 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-openshift-ca\") pod \"95ad37e2-90ac-4105-8175-4245b01a3bf7\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " Oct 30 06:14:22.851289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.851266 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ad37e2-90ac-4105-8175-4245b01a3bf7-kube-api-access-pv8qh\" (UniqueName: \"kubernetes.io/projected/95ad37e2-90ac-4105-8175-4245b01a3bf7-kube-api-access-pv8qh\") pod \"95ad37e2-90ac-4105-8175-4245b01a3bf7\" (UID: \"95ad37e2-90ac-4105-8175-4245b01a3bf7\") " Oct 30 06:14:22.852385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.852363 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:22.860766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.860732 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ad37e2-90ac-4105-8175-4245b01a3bf7-kube-api-access-pv8qh" pod "95ad37e2-90ac-4105-8175-4245b01a3bf7" (UID: "95ad37e2-90ac-4105-8175-4245b01a3bf7"). InnerVolumeSpecName "kube-api-access-pv8qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:22.881825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.881751 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-xks-ca" pod "95ad37e2-90ac-4105-8175-4245b01a3bf7" (UID: "95ad37e2-90ac-4105-8175-4245b01a3bf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:22.954705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.954668 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv8qh\" (UniqueName: \"kubernetes.io/projected/95ad37e2-90ac-4105-8175-4245b01a3bf7-kube-api-access-pv8qh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:22.954705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:22.954710 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ad37e2-90ac-4105-8175-4245b01a3bf7-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:23.252126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:23.252093 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" Oct 30 06:14:23.252126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:23.252110 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx" event={"ID":"95ad37e2-90ac-4105-8175-4245b01a3bf7","Type":"ContainerDied","Data":"98cff540bc03a67b13df4210c6fd13fe022dfc20b9f4af3ffc576eb72c4b7f93"} Oct 30 06:14:23.252710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:23.252198 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98cff540bc03a67b13df4210c6fd13fe022dfc20b9f4af3ffc576eb72c4b7f93" Oct 30 06:14:26.293296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.290492 5001 generic.go:356] "Generic (PLEG): container finished" podID="dc51944b-1255-4681-a6dc-fc6e503116b5" containerID="38c120a9d7ea02f0a2c3d955a262bb04ea8fccafa61502284760c8520ca5f798" exitCode=0 Oct 30 06:14:26.293296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.290584 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" event={"ID":"dc51944b-1255-4681-a6dc-fc6e503116b5","Type":"ContainerDied","Data":"38c120a9d7ea02f0a2c3d955a262bb04ea8fccafa61502284760c8520ca5f798"} Oct 30 06:14:26.293296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.292798 5001 generic.go:356] "Generic (PLEG): container finished" podID="142a47dc-0a88-4e03-b0b7-8a62d8b8e861" containerID="3c2b21d10b2791e311806d6a3c006ef85c358fb50a9b94adf80aaec1c7d33e7f" exitCode=0 Oct 30 06:14:26.293296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.292874 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" event={"ID":"142a47dc-0a88-4e03-b0b7-8a62d8b8e861","Type":"ContainerDied","Data":"3c2b21d10b2791e311806d6a3c006ef85c358fb50a9b94adf80aaec1c7d33e7f"} Oct 30 06:14:26.418550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.418522 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:14:26.463239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.463214 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:14:26.508228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.507712 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-plugins-config-volume\") pod \"142a47dc-0a88-4e03-b0b7-8a62d8b8e861\" (UID: \"142a47dc-0a88-4e03-b0b7-8a62d8b8e861\") " Oct 30 06:14:26.508795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.508495 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-kube-api-access-4s6tm\" (UniqueName: \"kubernetes.io/projected/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-kube-api-access-4s6tm\") pod \"142a47dc-0a88-4e03-b0b7-8a62d8b8e861\" (UID: \"142a47dc-0a88-4e03-b0b7-8a62d8b8e861\") " Oct 30 06:14:26.508795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.508593 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc51944b-1255-4681-a6dc-fc6e503116b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc51944b-1255-4681-a6dc-fc6e503116b5-plugins-config-volume\") pod \"dc51944b-1255-4681-a6dc-fc6e503116b5\" (UID: \"dc51944b-1255-4681-a6dc-fc6e503116b5\") " Oct 30 06:14:26.508795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.508649 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-plugins-config-volume" pod "142a47dc-0a88-4e03-b0b7-8a62d8b8e861" (UID: "142a47dc-0a88-4e03-b0b7-8a62d8b8e861"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:26.508795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.508702 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc51944b-1255-4681-a6dc-fc6e503116b5-kube-api-access-hjbrn\" (UniqueName: \"kubernetes.io/projected/dc51944b-1255-4681-a6dc-fc6e503116b5-kube-api-access-hjbrn\") pod \"dc51944b-1255-4681-a6dc-fc6e503116b5\" (UID: \"dc51944b-1255-4681-a6dc-fc6e503116b5\") " Oct 30 06:14:26.509806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.509781 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc51944b-1255-4681-a6dc-fc6e503116b5-plugins-config-volume" pod "dc51944b-1255-4681-a6dc-fc6e503116b5" (UID: "dc51944b-1255-4681-a6dc-fc6e503116b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:26.512381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.510538 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc51944b-1255-4681-a6dc-fc6e503116b5-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:26.512381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.510563 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:26.515941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.515917 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-kube-api-access-4s6tm" pod "142a47dc-0a88-4e03-b0b7-8a62d8b8e861" (UID: "142a47dc-0a88-4e03-b0b7-8a62d8b8e861"). InnerVolumeSpecName "kube-api-access-4s6tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:26.516050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.515954 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc51944b-1255-4681-a6dc-fc6e503116b5-kube-api-access-hjbrn" pod "dc51944b-1255-4681-a6dc-fc6e503116b5" (UID: "dc51944b-1255-4681-a6dc-fc6e503116b5"). InnerVolumeSpecName "kube-api-access-hjbrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:26.612256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.612210 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjbrn\" (UniqueName: \"kubernetes.io/projected/dc51944b-1255-4681-a6dc-fc6e503116b5-kube-api-access-hjbrn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:26.612256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:26.612252 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s6tm\" (UniqueName: \"kubernetes.io/projected/142a47dc-0a88-4e03-b0b7-8a62d8b8e861-kube-api-access-4s6tm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:27.339376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.338468 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" Oct 30 06:14:27.339376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.338548 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9" event={"ID":"dc51944b-1255-4681-a6dc-fc6e503116b5","Type":"ContainerDied","Data":"2500263933beeacfd362b79ca96e4af42d3fcb09eb3ef7c0e80e62b3606bc3cc"} Oct 30 06:14:27.339376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.338649 5001 scope.go:122] "RemoveContainer" containerID="38c120a9d7ea02f0a2c3d955a262bb04ea8fccafa61502284760c8520ca5f798" Oct 30 06:14:27.361169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.360237 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" event={"ID":"142a47dc-0a88-4e03-b0b7-8a62d8b8e861","Type":"ContainerDied","Data":"ecc79018cc4dfb459160efa9ce60d4137a8e2c7871dfd21b76b7c7076ae301fb"} Oct 30 06:14:27.361169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.360396 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh" Oct 30 06:14:27.372473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.372417 5001 scope.go:122] "RemoveContainer" containerID="3c2b21d10b2791e311806d6a3c006ef85c358fb50a9b94adf80aaec1c7d33e7f" Oct 30 06:14:27.420075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.420044 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9"] Oct 30 06:14:27.430029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.430004 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-pj7z9"] Oct 30 06:14:27.459444 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.459404 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh"] Oct 30 06:14:27.467200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.467174 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-kfvjh"] Oct 30 06:14:27.811424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.811391 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h"] Oct 30 06:14:27.811618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:27.811596 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" containerID="cri-o://39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4" gracePeriod=30 Oct 30 06:14:28.178331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.176603 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142a47dc-0a88-4e03-b0b7-8a62d8b8e861" path="/var/lib/kubelet/pods/142a47dc-0a88-4e03-b0b7-8a62d8b8e861/volumes" Oct 30 06:14:28.186341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.185193 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc51944b-1255-4681-a6dc-fc6e503116b5" path="/var/lib/kubelet/pods/dc51944b-1255-4681-a6dc-fc6e503116b5/volumes" Oct 30 06:14:28.448512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.448438 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf"] Oct 30 06:14:28.449120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.449106 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dc51944b-1255-4681-a6dc-fc6e503116b5" containerName="payload-pre-processing" Oct 30 06:14:28.449170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.449121 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc51944b-1255-4681-a6dc-fc6e503116b5" containerName="payload-pre-processing" Oct 30 06:14:28.449170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.449146 5001 cpu_manager.go:414] "containerMap: removing container" podUID="142a47dc-0a88-4e03-b0b7-8a62d8b8e861" containerName="payload-processing" Oct 30 06:14:28.449170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.449152 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142a47dc-0a88-4e03-b0b7-8a62d8b8e861" containerName="payload-processing" Oct 30 06:14:28.449268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.449182 5001 cpu_manager.go:414] "containerMap: removing container" podUID="95ad37e2-90ac-4105-8175-4245b01a3bf7" containerName="revoke-keys" Oct 30 06:14:28.449268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.449187 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ad37e2-90ac-4105-8175-4245b01a3bf7" containerName="revoke-keys" Oct 30 06:14:28.449370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.449291 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ad37e2-90ac-4105-8175-4245b01a3bf7" containerName="revoke-keys" Oct 30 06:14:28.449370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.449316 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc51944b-1255-4681-a6dc-fc6e503116b5" containerName="payload-pre-processing" Oct 30 06:14:28.449370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.449326 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="142a47dc-0a88-4e03-b0b7-8a62d8b8e861" containerName="payload-processing" Oct 30 06:14:28.453017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.453001 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:28.468518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.468493 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:14:28.476334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.476282 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf"] Oct 30 06:14:28.482764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.480442 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:14:28.482764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.480473 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:14:28.482764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.480630 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:14:28.593966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.593931 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f789d7-qffpf\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:28.594126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.594078 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsww\" (UniqueName: \"kubernetes.io/projected/68a62a74-8357-4261-9f6c-2da2f388d02f-kube-api-access-zhsww\") pod \"maas-api-revoke-keys-e2e-api-b-f789d7-qffpf\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:28.594205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.594179 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f789d7-qffpf\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:28.696534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.696489 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsww\" (UniqueName: \"kubernetes.io/projected/68a62a74-8357-4261-9f6c-2da2f388d02f-kube-api-access-zhsww\") pod \"maas-api-revoke-keys-e2e-api-b-f789d7-qffpf\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:28.696723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.696609 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f789d7-qffpf\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:28.696894 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.696877 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f789d7-qffpf\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:28.697643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.697614 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f789d7-qffpf\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:28.699238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.699178 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f789d7-qffpf\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:28.720452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.720424 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsww\" (UniqueName: \"kubernetes.io/projected/68a62a74-8357-4261-9f6c-2da2f388d02f-kube-api-access-zhsww\") pod \"maas-api-revoke-keys-e2e-api-b-f789d7-qffpf\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:28.771024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.770989 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:28.956785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.956700 5001 patch_prober.go:29] interesting pod/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:14:28.956785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:28.956765 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:14:29.088117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:29.088080 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf"] Oct 30 06:14:29.092118 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:14:29.092085 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a62a74_8357_4261_9f6c_2da2f388d02f.slice/crio-321404294be348b9784028cc5af403307b43736852e78279e417432d8290daa2 WatchSource:0}: Error finding container 321404294be348b9784028cc5af403307b43736852e78279e417432d8290daa2: Status 404 returned error can't find the container with id 321404294be348b9784028cc5af403307b43736852e78279e417432d8290daa2 Oct 30 06:14:29.213781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:29.213699 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:14:29.228328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:29.227843 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:14:29.394646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:29.394616 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" event={"ID":"68a62a74-8357-4261-9f6c-2da2f388d02f","Type":"ContainerStarted","Data":"321404294be348b9784028cc5af403307b43736852e78279e417432d8290daa2"} Oct 30 06:14:30.035096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:30.035065 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd"] Oct 30 06:14:30.048151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:30.048119 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca0315c1-hl7vd"] Oct 30 06:14:30.121490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:30.121453 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73cb86f-83d8-492b-8a2a-41027668e826" path="/var/lib/kubelet/pods/c73cb86f-83d8-492b-8a2a-41027668e826/volumes" Oct 30 06:14:30.406610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:30.406580 5001 generic.go:356] "Generic (PLEG): container finished" podID="68a62a74-8357-4261-9f6c-2da2f388d02f" containerID="54d3ffbf5ab0fad768ecb225e95b3e2e3e50e71b0349b8094f1e4db092cab05c" exitCode=0 Oct 30 06:14:30.406837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:30.406675 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" event={"ID":"68a62a74-8357-4261-9f6c-2da2f388d02f","Type":"ContainerDied","Data":"54d3ffbf5ab0fad768ecb225e95b3e2e3e50e71b0349b8094f1e4db092cab05c"} Oct 30 06:14:31.582635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:31.582611 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:31.653272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:31.653239 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-xks-ca\") pod \"68a62a74-8357-4261-9f6c-2da2f388d02f\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " Oct 30 06:14:31.653510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:31.653361 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-openshift-ca\") pod \"68a62a74-8357-4261-9f6c-2da2f388d02f\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " Oct 30 06:14:31.653574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:31.653509 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a62a74-8357-4261-9f6c-2da2f388d02f-kube-api-access-zhsww\" (UniqueName: \"kubernetes.io/projected/68a62a74-8357-4261-9f6c-2da2f388d02f-kube-api-access-zhsww\") pod \"68a62a74-8357-4261-9f6c-2da2f388d02f\" (UID: \"68a62a74-8357-4261-9f6c-2da2f388d02f\") " Oct 30 06:14:31.655997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:31.655940 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-openshift-ca" pod "68a62a74-8357-4261-9f6c-2da2f388d02f" (UID: "68a62a74-8357-4261-9f6c-2da2f388d02f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:31.657535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:31.657503 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-xks-ca" pod "68a62a74-8357-4261-9f6c-2da2f388d02f" (UID: "68a62a74-8357-4261-9f6c-2da2f388d02f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:31.658594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:31.658563 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a62a74-8357-4261-9f6c-2da2f388d02f-kube-api-access-zhsww" pod "68a62a74-8357-4261-9f6c-2da2f388d02f" (UID: "68a62a74-8357-4261-9f6c-2da2f388d02f"). InnerVolumeSpecName "kube-api-access-zhsww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:31.756104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:31.756069 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:31.756104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:31.756099 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a62a74-8357-4261-9f6c-2da2f388d02f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:31.756104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:31.756110 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsww\" (UniqueName: \"kubernetes.io/projected/68a62a74-8357-4261-9f6c-2da2f388d02f-kube-api-access-zhsww\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:32.443431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:32.442716 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" event={"ID":"68a62a74-8357-4261-9f6c-2da2f388d02f","Type":"ContainerDied","Data":"321404294be348b9784028cc5af403307b43736852e78279e417432d8290daa2"} Oct 30 06:14:32.443431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:32.442779 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="321404294be348b9784028cc5af403307b43736852e78279e417432d8290daa2" Oct 30 06:14:32.443431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:32.442855 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf" Oct 30 06:14:33.204514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.204483 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:14:33.302020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.301982 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-data\") pod \"c0d29e78-7d62-4e08-b297-9bef59250b4c\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " Oct 30 06:14:33.302020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.302033 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-certs\") pod \"c0d29e78-7d62-4e08-b297-9bef59250b4c\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " Oct 30 06:14:33.302274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.302106 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-envoy\") pod \"c0d29e78-7d62-4e08-b297-9bef59250b4c\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " Oct 30 06:14:33.302274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.302139 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d29e78-7d62-4e08-b297-9bef59250b4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d29e78-7d62-4e08-b297-9bef59250b4c-istiod-ca-cert\") pod \"c0d29e78-7d62-4e08-b297-9bef59250b4c\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " Oct 30 06:14:33.302406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.302274 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-token\" (UniqueName: \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-token\") pod \"c0d29e78-7d62-4e08-b297-9bef59250b4c\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " Oct 30 06:14:33.302406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.302329 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-data" pod "c0d29e78-7d62-4e08-b297-9bef59250b4c" (UID: "c0d29e78-7d62-4e08-b297-9bef59250b4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:33.302406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.302395 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-kube-api-access-jgfpr\" (UniqueName: \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-kube-api-access-jgfpr\") pod \"c0d29e78-7d62-4e08-b297-9bef59250b4c\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " Oct 30 06:14:33.302564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.302444 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-credential-socket\") pod \"c0d29e78-7d62-4e08-b297-9bef59250b4c\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " Oct 30 06:14:33.302564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.302551 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-socket\") pod \"c0d29e78-7d62-4e08-b297-9bef59250b4c\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " Oct 30 06:14:33.302663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.302640 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-podinfo\") pod \"c0d29e78-7d62-4e08-b297-9bef59250b4c\" (UID: \"c0d29e78-7d62-4e08-b297-9bef59250b4c\") " Oct 30 06:14:33.302817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.302780 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d29e78-7d62-4e08-b297-9bef59250b4c-istiod-ca-cert" pod "c0d29e78-7d62-4e08-b297-9bef59250b4c" (UID: "c0d29e78-7d62-4e08-b297-9bef59250b4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:33.303369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.303284 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-credential-socket" pod "c0d29e78-7d62-4e08-b297-9bef59250b4c" (UID: "c0d29e78-7d62-4e08-b297-9bef59250b4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:33.303369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.303320 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-certs" pod "c0d29e78-7d62-4e08-b297-9bef59250b4c" (UID: "c0d29e78-7d62-4e08-b297-9bef59250b4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:33.303735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.303520 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-socket" pod "c0d29e78-7d62-4e08-b297-9bef59250b4c" (UID: "c0d29e78-7d62-4e08-b297-9bef59250b4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:33.304160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.304140 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:33.304160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.304160 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:33.304359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.304170 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:33.304359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.304178 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:33.304359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.304187 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d29e78-7d62-4e08-b297-9bef59250b4c-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:33.309351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.306379 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-podinfo" pod "c0d29e78-7d62-4e08-b297-9bef59250b4c" (UID: "c0d29e78-7d62-4e08-b297-9bef59250b4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:14:33.309351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.306672 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-kube-api-access-jgfpr" pod "c0d29e78-7d62-4e08-b297-9bef59250b4c" (UID: "c0d29e78-7d62-4e08-b297-9bef59250b4c"). InnerVolumeSpecName "kube-api-access-jgfpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:33.309351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.306686 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-token" pod "c0d29e78-7d62-4e08-b297-9bef59250b4c" (UID: "c0d29e78-7d62-4e08-b297-9bef59250b4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:33.309351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.307081 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-envoy" pod "c0d29e78-7d62-4e08-b297-9bef59250b4c" (UID: "c0d29e78-7d62-4e08-b297-9bef59250b4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:33.405614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.405525 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:33.405614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.405552 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:33.405614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.405565 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:33.405614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.405577 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgfpr\" (UniqueName: \"kubernetes.io/projected/c0d29e78-7d62-4e08-b297-9bef59250b4c-kube-api-access-jgfpr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:33.453240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.453208 5001 generic.go:356] "Generic (PLEG): container finished" podID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerID="39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4" exitCode=0 Oct 30 06:14:33.453461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.453287 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" Oct 30 06:14:33.453461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.453356 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" event={"ID":"c0d29e78-7d62-4e08-b297-9bef59250b4c","Type":"ContainerDied","Data":"39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4"} Oct 30 06:14:33.453461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.453401 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h" event={"ID":"c0d29e78-7d62-4e08-b297-9bef59250b4c","Type":"ContainerDied","Data":"90dfca69954a20db909b99238e0d463078b8c6e553eb27eebd52ff1a471763a6"} Oct 30 06:14:33.453461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.453422 5001 scope.go:122] "RemoveContainer" containerID="39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4" Oct 30 06:14:33.465452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.465431 5001 scope.go:122] "RemoveContainer" containerID="17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1" Oct 30 06:14:33.475735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.475714 5001 scope.go:122] "RemoveContainer" containerID="39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4" Oct 30 06:14:33.476077 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:14:33.476052 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4\": container with ID starting with 39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4 not found: ID does not exist" containerID="39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4" Oct 30 06:14:33.476134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.476083 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4"} err="failed to get container status \"39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4\": rpc error: code = NotFound desc = could not find container \"39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4\": container with ID starting with 39ec9a2bed99e9938b278ee5bc3f3e9e18bc4891f418bde8aee26c4e714d80e4 not found: ID does not exist" Oct 30 06:14:33.476134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.476112 5001 scope.go:122] "RemoveContainer" containerID="17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1" Oct 30 06:14:33.476393 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:14:33.476370 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1\": container with ID starting with 17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1 not found: ID does not exist" containerID="17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1" Oct 30 06:14:33.476450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.476405 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1"} err="failed to get container status \"17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1\": rpc error: code = NotFound desc = could not find container \"17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1\": container with ID starting with 17ba07c4f53706196119bf54bb1aafc65056daa8c5799fcd70922026081291e1 not found: ID does not exist" Oct 30 06:14:33.514193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.514168 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h"] Oct 30 06:14:33.518968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:33.518947 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1ec766-openshift-default-69f8887585-55l2h"] Oct 30 06:14:34.120601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:34.120568 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" path="/var/lib/kubelet/pods/c0d29e78-7d62-4e08-b297-9bef59250b4c/volumes" Oct 30 06:14:37.054613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:37.054581 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4"] Oct 30 06:14:37.067833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:37.067805 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-4rpd4"] Oct 30 06:14:38.119072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:38.119038 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dbf896a-d653-4437-b7d5-fb4ea05712e6" path="/var/lib/kubelet/pods/7dbf896a-d653-4437-b7d5-fb4ea05712e6/volumes" Oct 30 06:14:38.270359 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:14:38.270269 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:14:38.896807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:38.896773 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j"] Oct 30 06:14:38.943325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:38.943280 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8"] Oct 30 06:14:43.897190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:43.897148 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" podUID="ddbaa0c1-73a6-473a-89e6-79fb010aa88d" containerName="payload-processing" containerID="cri-o://2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98" gracePeriod=25 Oct 30 06:14:43.944278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:43.944223 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" podUID="4175ac21-794a-4a88-9305-926195379728" containerName="payload-pre-processing" containerID="cri-o://b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e" gracePeriod=25 Oct 30 06:14:44.299938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.299903 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:14:44.321263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.321243 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:14:44.366810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.366780 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-kube-api-access-62q6s\" (UniqueName: \"kubernetes.io/projected/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-kube-api-access-62q6s\") pod \"ddbaa0c1-73a6-473a-89e6-79fb010aa88d\" (UID: \"ddbaa0c1-73a6-473a-89e6-79fb010aa88d\") " Oct 30 06:14:44.367183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.367151 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4175ac21-794a-4a88-9305-926195379728-kube-api-access-6zdzv\" (UniqueName: \"kubernetes.io/projected/4175ac21-794a-4a88-9305-926195379728-kube-api-access-6zdzv\") pod \"4175ac21-794a-4a88-9305-926195379728\" (UID: \"4175ac21-794a-4a88-9305-926195379728\") " Oct 30 06:14:44.367511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.367492 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-plugins-config-volume\") pod \"ddbaa0c1-73a6-473a-89e6-79fb010aa88d\" (UID: \"ddbaa0c1-73a6-473a-89e6-79fb010aa88d\") " Oct 30 06:14:44.367752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.367734 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4175ac21-794a-4a88-9305-926195379728-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4175ac21-794a-4a88-9305-926195379728-plugins-config-volume\") pod \"4175ac21-794a-4a88-9305-926195379728\" (UID: \"4175ac21-794a-4a88-9305-926195379728\") " Oct 30 06:14:44.368178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.368010 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-plugins-config-volume" pod "ddbaa0c1-73a6-473a-89e6-79fb010aa88d" (UID: "ddbaa0c1-73a6-473a-89e6-79fb010aa88d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:44.368272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.368191 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4175ac21-794a-4a88-9305-926195379728-plugins-config-volume" pod "4175ac21-794a-4a88-9305-926195379728" (UID: "4175ac21-794a-4a88-9305-926195379728"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:44.369196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.369165 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4175ac21-794a-4a88-9305-926195379728-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:44.369196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.369194 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:44.370827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.370806 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4175ac21-794a-4a88-9305-926195379728-kube-api-access-6zdzv" pod "4175ac21-794a-4a88-9305-926195379728" (UID: "4175ac21-794a-4a88-9305-926195379728"). InnerVolumeSpecName "kube-api-access-6zdzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:44.373137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.373101 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-kube-api-access-62q6s" pod "ddbaa0c1-73a6-473a-89e6-79fb010aa88d" (UID: "ddbaa0c1-73a6-473a-89e6-79fb010aa88d"). InnerVolumeSpecName "kube-api-access-62q6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:44.471239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.471155 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62q6s\" (UniqueName: \"kubernetes.io/projected/ddbaa0c1-73a6-473a-89e6-79fb010aa88d-kube-api-access-62q6s\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:44.471239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.471192 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zdzv\" (UniqueName: \"kubernetes.io/projected/4175ac21-794a-4a88-9305-926195379728-kube-api-access-6zdzv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:14:44.734242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.734144 5001 generic.go:356] "Generic (PLEG): container finished" podID="ddbaa0c1-73a6-473a-89e6-79fb010aa88d" containerID="2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98" exitCode=0 Oct 30 06:14:44.734242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.734189 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" Oct 30 06:14:44.734481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.734238 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" event={"ID":"ddbaa0c1-73a6-473a-89e6-79fb010aa88d","Type":"ContainerDied","Data":"2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98"} Oct 30 06:14:44.734481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.734334 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j" event={"ID":"ddbaa0c1-73a6-473a-89e6-79fb010aa88d","Type":"ContainerDied","Data":"8230490debc6cce3cabc8b0b83757b84399b1167db955541ac43e7a3d751c10a"} Oct 30 06:14:44.734481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.734346 5001 scope.go:122] "RemoveContainer" containerID="2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98" Oct 30 06:14:44.735983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.735959 5001 generic.go:356] "Generic (PLEG): container finished" podID="4175ac21-794a-4a88-9305-926195379728" containerID="b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e" exitCode=0 Oct 30 06:14:44.736107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.736022 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" Oct 30 06:14:44.736107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.736094 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" event={"ID":"4175ac21-794a-4a88-9305-926195379728","Type":"ContainerDied","Data":"b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e"} Oct 30 06:14:44.736221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.736129 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8" event={"ID":"4175ac21-794a-4a88-9305-926195379728","Type":"ContainerDied","Data":"a8772a35c59a5c9481dfe53275242885394ee867245786fd8673deeb8f0538a2"} Oct 30 06:14:44.746615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.746598 5001 scope.go:122] "RemoveContainer" containerID="2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98" Oct 30 06:14:44.746897 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:14:44.746874 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98\": container with ID starting with 2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98 not found: ID does not exist" containerID="2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98" Oct 30 06:14:44.746968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.746904 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98"} err="failed to get container status \"2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98\": rpc error: code = NotFound desc = could not find container \"2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98\": container with ID starting with 2ef42d0c568d60f9514f8be851e45e9e31e7a1895a064dae5f4e29e2f692ac98 not found: ID does not exist" Oct 30 06:14:44.746968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.746920 5001 scope.go:122] "RemoveContainer" containerID="b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e" Oct 30 06:14:44.759353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.759336 5001 scope.go:122] "RemoveContainer" containerID="b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e" Oct 30 06:14:44.759644 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:14:44.759621 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e\": container with ID starting with b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e not found: ID does not exist" containerID="b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e" Oct 30 06:14:44.759720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.759655 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e"} err="failed to get container status \"b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e\": rpc error: code = NotFound desc = could not find container \"b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e\": container with ID starting with b883fb2a641c12b760c5ffbb13408e1ac3bce262bd1ff10dc6c1fc94dacf138e not found: ID does not exist" Oct 30 06:14:44.790608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.790585 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j"] Oct 30 06:14:44.795859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.795839 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f789d7-84df74b86b-tqn5j"] Oct 30 06:14:44.832676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.832650 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8"] Oct 30 06:14:44.836997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:44.836967 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f789d7-7795658f8c-4h5s8"] Oct 30 06:14:46.382550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:46.382510 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4175ac21-794a-4a88-9305-926195379728" path="/var/lib/kubelet/pods/4175ac21-794a-4a88-9305-926195379728/volumes" Oct 30 06:14:46.383876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:46.383851 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddbaa0c1-73a6-473a-89e6-79fb010aa88d" path="/var/lib/kubelet/pods/ddbaa0c1-73a6-473a-89e6-79fb010aa88d/volumes" Oct 30 06:14:46.390127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:46.390107 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:14:50.095333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:50.092411 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nnt77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:14:50.095333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:50.092483 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:14:51.002188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.002146 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26"] Oct 30 06:14:51.002902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.002883 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ddbaa0c1-73a6-473a-89e6-79fb010aa88d" containerName="payload-processing" Oct 30 06:14:51.002902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.002901 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbaa0c1-73a6-473a-89e6-79fb010aa88d" containerName="payload-processing" Oct 30 06:14:51.003042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.002916 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" Oct 30 06:14:51.003042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.002924 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" Oct 30 06:14:51.003042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.002943 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" Oct 30 06:14:51.003042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.002950 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" Oct 30 06:14:51.003042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.002961 5001 cpu_manager.go:414] "containerMap: removing container" podUID="68a62a74-8357-4261-9f6c-2da2f388d02f" containerName="revoke-keys" Oct 30 06:14:51.003042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.002965 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a62a74-8357-4261-9f6c-2da2f388d02f" containerName="revoke-keys" Oct 30 06:14:51.003042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.002991 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4175ac21-794a-4a88-9305-926195379728" containerName="payload-pre-processing" Oct 30 06:14:51.003042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.002995 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4175ac21-794a-4a88-9305-926195379728" containerName="payload-pre-processing" Oct 30 06:14:51.003290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.003098 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4175ac21-794a-4a88-9305-926195379728" containerName="payload-pre-processing" Oct 30 06:14:51.003290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.003111 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbaa0c1-73a6-473a-89e6-79fb010aa88d" containerName="payload-processing" Oct 30 06:14:51.003290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.003119 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" Oct 30 06:14:51.003290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.003126 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d29e78-7d62-4e08-b297-9bef59250b4c" containerName="istio-proxy" Oct 30 06:14:51.003290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.003132 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a62a74-8357-4261-9f6c-2da2f388d02f" containerName="revoke-keys" Oct 30 06:14:51.008614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.008594 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:14:51.035896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.035862 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26"] Oct 30 06:14:51.036244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.036217 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ba44b248-dockercfg-9k2rv\"" Oct 30 06:14:51.036563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.036547 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ba44b248\"" Oct 30 06:14:51.125367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.120455 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmpln\" (UniqueName: \"kubernetes.io/projected/640bf110-4c9a-477f-bb8e-fb32739309ce-kube-api-access-cmpln\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26\" (UID: \"640bf110-4c9a-477f-bb8e-fb32739309ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:14:51.125367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.120539 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640bf110-4c9a-477f-bb8e-fb32739309ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26\" (UID: \"640bf110-4c9a-477f-bb8e-fb32739309ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:14:51.224502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.224456 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmpln\" (UniqueName: \"kubernetes.io/projected/640bf110-4c9a-477f-bb8e-fb32739309ce-kube-api-access-cmpln\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26\" (UID: \"640bf110-4c9a-477f-bb8e-fb32739309ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:14:51.239836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.239802 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640bf110-4c9a-477f-bb8e-fb32739309ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26\" (UID: \"640bf110-4c9a-477f-bb8e-fb32739309ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:14:51.258807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.225850 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc"] Oct 30 06:14:51.274456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.258069 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640bf110-4c9a-477f-bb8e-fb32739309ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26\" (UID: \"640bf110-4c9a-477f-bb8e-fb32739309ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:14:51.343126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.298321 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:14:51.410343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.363883 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7plph\" (UniqueName: \"kubernetes.io/projected/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-kube-api-access-7plph\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-2z4nc\" (UID: \"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:14:51.450407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.383251 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc"] Oct 30 06:14:51.450407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.449626 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-plugins-config-volume\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-2z4nc\" (UID: \"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:14:51.480328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.468259 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmpln\" (UniqueName: \"kubernetes.io/projected/640bf110-4c9a-477f-bb8e-fb32739309ce-kube-api-access-cmpln\") pod \"payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26\" (UID: \"640bf110-4c9a-477f-bb8e-fb32739309ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:14:51.551222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.551140 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7plph\" (UniqueName: \"kubernetes.io/projected/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-kube-api-access-7plph\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-2z4nc\" (UID: \"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:14:51.551222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.551202 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-plugins-config-volume\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-2z4nc\" (UID: \"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:14:51.551964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.551942 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-plugins-config-volume\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-2z4nc\" (UID: \"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:14:51.576044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.576010 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7plph\" (UniqueName: \"kubernetes.io/projected/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-kube-api-access-7plph\") pod \"payload-processing-e2e-mt-ba44b248-868467698b-2z4nc\" (UID: \"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e\") " pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:14:51.625504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.625481 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:14:51.744386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:51.744350 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:14:52.024605 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:14:52.024481 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640bf110_4c9a_477f_bb8e_fb32739309ce.slice/crio-64d235602e8275e59c47d3057595f80d07a582c3e87fc9721ae6c5763ff3ec5e WatchSource:0}: Error finding container 64d235602e8275e59c47d3057595f80d07a582c3e87fc9721ae6c5763ff3ec5e: Status 404 returned error can't find the container with id 64d235602e8275e59c47d3057595f80d07a582c3e87fc9721ae6c5763ff3ec5e Oct 30 06:14:52.099388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:52.092993 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26"] Oct 30 06:14:52.165192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:52.165154 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc"] Oct 30 06:14:52.891170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:52.891014 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" event={"ID":"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e","Type":"ContainerStarted","Data":"d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982"} Oct 30 06:14:52.891170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:52.891085 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" event={"ID":"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e","Type":"ContainerStarted","Data":"099a9a4583ceaec6c4107f96b801274f50dfc8ed42e9fef9c496537e244d5430"} Oct 30 06:14:52.891606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:52.891561 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:14:52.893340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:52.893296 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" event={"ID":"640bf110-4c9a-477f-bb8e-fb32739309ce","Type":"ContainerStarted","Data":"8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37"} Oct 30 06:14:52.893526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:52.893506 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" event={"ID":"640bf110-4c9a-477f-bb8e-fb32739309ce","Type":"ContainerStarted","Data":"64d235602e8275e59c47d3057595f80d07a582c3e87fc9721ae6c5763ff3ec5e"} Oct 30 06:14:52.893666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:52.893654 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:14:52.935077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:52.935028 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" podStartSLOduration=1.935016467 podStartE2EDuration="1.935016467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:52.933502161 +0000 UTC m=+1306.929459185" watchObservedRunningTime="2026-10-30 06:14:52.935016467 +0000 UTC m=+1306.930973477" Oct 30 06:14:56.712761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:56.712689 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" podStartSLOduration=6.71267103 podStartE2EDuration="6.71267103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:52.98607681 +0000 UTC m=+1306.982033820" watchObservedRunningTime="2026-10-30 06:14:56.71267103 +0000 UTC m=+1310.708628040" Oct 30 06:14:56.717400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:56.717374 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb"] Oct 30 06:14:56.721822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:56.721799 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:14:56.748100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:56.748076 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb"] Oct 30 06:14:56.802121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:56.798723 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:14:56.802121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:56.798876 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:14:56.802121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:56.798939 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:14:56.802121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:56.799116 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:14:56.901841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:56.901784 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6df62-96mrb\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:14:56.902083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:56.901939 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6df62-96mrb\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:14:56.902083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:56.901984 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtzj6\" (UniqueName: \"kubernetes.io/projected/d3164cd0-0181-4645-9332-f82110a5e5c9-kube-api-access-vtzj6\") pod \"maas-api-revoke-keys-e2e-api-a-e6df62-96mrb\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:14:57.003511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:57.003406 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6df62-96mrb\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:14:57.003511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:57.003496 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6df62-96mrb\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:14:57.003764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:57.003528 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtzj6\" (UniqueName: \"kubernetes.io/projected/d3164cd0-0181-4645-9332-f82110a5e5c9-kube-api-access-vtzj6\") pod \"maas-api-revoke-keys-e2e-api-a-e6df62-96mrb\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:14:57.004227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:57.004200 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6df62-96mrb\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:14:57.005983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:57.005964 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6df62-96mrb\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:14:57.028219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:57.028194 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtzj6\" (UniqueName: \"kubernetes.io/projected/d3164cd0-0181-4645-9332-f82110a5e5c9-kube-api-access-vtzj6\") pod \"maas-api-revoke-keys-e2e-api-a-e6df62-96mrb\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:14:57.039219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:57.039192 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:14:57.421327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:57.420905 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb"] Oct 30 06:14:57.429585 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:14:57.429557 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3164cd0_0181_4645_9332_f82110a5e5c9.slice/crio-0193716c5c69c6a8ec0083196b794006282f71e5544d30b5d6e43aaf145f66e3 WatchSource:0}: Error finding container 0193716c5c69c6a8ec0083196b794006282f71e5544d30b5d6e43aaf145f66e3: Status 404 returned error can't find the container with id 0193716c5c69c6a8ec0083196b794006282f71e5544d30b5d6e43aaf145f66e3 Oct 30 06:14:57.943688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:57.943653 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" event={"ID":"d3164cd0-0181-4645-9332-f82110a5e5c9","Type":"ContainerStarted","Data":"0193716c5c69c6a8ec0083196b794006282f71e5544d30b5d6e43aaf145f66e3"} Oct 30 06:14:58.955902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:58.955811 5001 generic.go:356] "Generic (PLEG): container finished" podID="d3164cd0-0181-4645-9332-f82110a5e5c9" containerID="f87f5f263bbbbdae80e5573f320b36928a09141a72b54830ac14fbbf130e1479" exitCode=0 Oct 30 06:14:58.956289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:14:58.955904 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" event={"ID":"d3164cd0-0181-4645-9332-f82110a5e5c9","Type":"ContainerDied","Data":"f87f5f263bbbbdae80e5573f320b36928a09141a72b54830ac14fbbf130e1479"} Oct 30 06:15:00.168589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.168548 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:15:00.177757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.177730 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql"] Oct 30 06:15:00.178671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.178632 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d3164cd0-0181-4645-9332-f82110a5e5c9" containerName="revoke-keys" Oct 30 06:15:00.178671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.178659 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3164cd0-0181-4645-9332-f82110a5e5c9" containerName="revoke-keys" Oct 30 06:15:00.178922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.178906 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3164cd0-0181-4645-9332-f82110a5e5c9" containerName="revoke-keys" Oct 30 06:15:00.182837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.182816 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" Oct 30 06:15:00.184028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.184004 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f"] Oct 30 06:15:00.188459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.188419 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" Oct 30 06:15:00.188709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.188682 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q"] Oct 30 06:15:00.203533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.203179 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-xks-ca\") pod \"d3164cd0-0181-4645-9332-f82110a5e5c9\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " Oct 30 06:15:00.203533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.203501 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3164cd0-0181-4645-9332-f82110a5e5c9-kube-api-access-vtzj6\" (UniqueName: \"kubernetes.io/projected/d3164cd0-0181-4645-9332-f82110a5e5c9-kube-api-access-vtzj6\") pod \"d3164cd0-0181-4645-9332-f82110a5e5c9\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " Oct 30 06:15:00.203679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.203562 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-openshift-ca\") pod \"d3164cd0-0181-4645-9332-f82110a5e5c9\" (UID: \"d3164cd0-0181-4645-9332-f82110a5e5c9\") " Oct 30 06:15:00.205697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.205376 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-openshift-ca" pod "d3164cd0-0181-4645-9332-f82110a5e5c9" (UID: "d3164cd0-0181-4645-9332-f82110a5e5c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:00.209954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.208847 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr"] Oct 30 06:15:00.222496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.217389 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" Oct 30 06:15:00.222496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.218099 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-xks-ca" pod "d3164cd0-0181-4645-9332-f82110a5e5c9" (UID: "d3164cd0-0181-4645-9332-f82110a5e5c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:00.222496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.218238 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" Oct 30 06:15:00.225398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.225294 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f"] Oct 30 06:15:00.242256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.231134 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3164cd0-0181-4645-9332-f82110a5e5c9-kube-api-access-vtzj6" pod "d3164cd0-0181-4645-9332-f82110a5e5c9" (UID: "d3164cd0-0181-4645-9332-f82110a5e5c9"). InnerVolumeSpecName "kube-api-access-vtzj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:00.242256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.235728 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q"] Oct 30 06:15:00.248357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.248323 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr"] Oct 30 06:15:00.269990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.258282 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql"] Oct 30 06:15:00.269990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.264419 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc"] Oct 30 06:15:00.271695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.271021 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" Oct 30 06:15:00.293995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.292527 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z"] Oct 30 06:15:00.300408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.300252 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.306036 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b87dd6d-e591-475a-80b4-389052f0c713-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q\" (UID: \"5b87dd6d-e591-475a-80b4-389052f0c713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.306110 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw674\" (UniqueName: \"kubernetes.io/projected/6af42bac-496e-4ffa-92a3-64b7c4852a73-kube-api-access-kw674\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f\" (UID: \"6af42bac-496e-4ffa-92a3-64b7c4852a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.306148 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79946c52-915f-4a02-bce3-3d4c92c8360b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc\" (UID: \"79946c52-915f-4a02-bce3-3d4c92c8360b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.306185 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfgvv\" (UniqueName: \"kubernetes.io/projected/67ba38f1-1102-4702-b073-f80c906b1363-kube-api-access-lfgvv\") pod \"maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql\" (UID: \"67ba38f1-1102-4702-b073-f80c906b1363\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.306212 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrzp\" (UniqueName: \"kubernetes.io/projected/79946c52-915f-4a02-bce3-3d4c92c8360b-kube-api-access-qzrzp\") pod \"maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc\" (UID: \"79946c52-915f-4a02-bce3-3d4c92c8360b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.306444 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b4a966-5447-45de-ab4a-c79b929a3d78-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889015-x9tnr\" (UID: \"43b4a966-5447-45de-ab4a-c79b929a3d78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.306616 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af42bac-496e-4ffa-92a3-64b7c4852a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f\" (UID: \"6af42bac-496e-4ffa-92a3-64b7c4852a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.306752 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w8zf\" (UniqueName: \"kubernetes.io/projected/43b4a966-5447-45de-ab4a-c79b929a3d78-kube-api-access-9w8zf\") pod \"maas-api-key-cleanup-29889015-x9tnr\" (UID: \"43b4a966-5447-45de-ab4a-c79b929a3d78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.306825 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ba38f1-1102-4702-b073-f80c906b1363-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql\" (UID: \"67ba38f1-1102-4702-b073-f80c906b1363\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.306854 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wjj\" (UniqueName: \"kubernetes.io/projected/5b87dd6d-e591-475a-80b4-389052f0c713-kube-api-access-m7wjj\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q\" (UID: \"5b87dd6d-e591-475a-80b4-389052f0c713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.307039 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtzj6\" (UniqueName: \"kubernetes.io/projected/d3164cd0-0181-4645-9332-f82110a5e5c9-kube-api-access-vtzj6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.307056 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:00.308001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.307070 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3164cd0-0181-4645-9332-f82110a5e5c9-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:00.326842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.326813 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9"] Oct 30 06:15:00.332770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.332741 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" Oct 30 06:15:00.335697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.335626 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z"] Oct 30 06:15:00.347244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.346546 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc"] Oct 30 06:15:00.358980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.358520 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9"] Oct 30 06:15:00.408817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.408772 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w8zf\" (UniqueName: \"kubernetes.io/projected/43b4a966-5447-45de-ab4a-c79b929a3d78-kube-api-access-9w8zf\") pod \"maas-api-key-cleanup-29889015-x9tnr\" (UID: \"43b4a966-5447-45de-ab4a-c79b929a3d78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" Oct 30 06:15:00.409018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.408836 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ba38f1-1102-4702-b073-f80c906b1363-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql\" (UID: \"67ba38f1-1102-4702-b073-f80c906b1363\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" Oct 30 06:15:00.409018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.408867 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wjj\" (UniqueName: \"kubernetes.io/projected/5b87dd6d-e591-475a-80b4-389052f0c713-kube-api-access-m7wjj\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q\" (UID: \"5b87dd6d-e591-475a-80b4-389052f0c713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" Oct 30 06:15:00.409018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.408963 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b87dd6d-e591-475a-80b4-389052f0c713-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q\" (UID: \"5b87dd6d-e591-475a-80b4-389052f0c713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" Oct 30 06:15:00.409173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.409016 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw674\" (UniqueName: \"kubernetes.io/projected/6af42bac-496e-4ffa-92a3-64b7c4852a73-kube-api-access-kw674\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f\" (UID: \"6af42bac-496e-4ffa-92a3-64b7c4852a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" Oct 30 06:15:00.409173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.409049 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79946c52-915f-4a02-bce3-3d4c92c8360b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc\" (UID: \"79946c52-915f-4a02-bce3-3d4c92c8360b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" Oct 30 06:15:00.409173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.409083 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgvv\" (UniqueName: \"kubernetes.io/projected/67ba38f1-1102-4702-b073-f80c906b1363-kube-api-access-lfgvv\") pod \"maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql\" (UID: \"67ba38f1-1102-4702-b073-f80c906b1363\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" Oct 30 06:15:00.409173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.409121 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrzp\" (UniqueName: \"kubernetes.io/projected/79946c52-915f-4a02-bce3-3d4c92c8360b-kube-api-access-qzrzp\") pod \"maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc\" (UID: \"79946c52-915f-4a02-bce3-3d4c92c8360b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" Oct 30 06:15:00.409173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.409167 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9\" (UID: \"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" Oct 30 06:15:00.409482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.409254 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmgvw\" (UniqueName: \"kubernetes.io/projected/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-kube-api-access-dmgvw\") pod \"maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9\" (UID: \"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" Oct 30 06:15:00.409482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.409284 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b4a966-5447-45de-ab4a-c79b929a3d78-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889015-x9tnr\" (UID: \"43b4a966-5447-45de-ab4a-c79b929a3d78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" Oct 30 06:15:00.409482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.409350 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fctxx\" (UniqueName: \"kubernetes.io/projected/592c1065-0e61-4095-8234-5cba81dd1283-kube-api-access-fctxx\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z\" (UID: \"592c1065-0e61-4095-8234-5cba81dd1283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" Oct 30 06:15:00.409482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.409433 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/592c1065-0e61-4095-8234-5cba81dd1283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z\" (UID: \"592c1065-0e61-4095-8234-5cba81dd1283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" Oct 30 06:15:00.409693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.409526 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af42bac-496e-4ffa-92a3-64b7c4852a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f\" (UID: \"6af42bac-496e-4ffa-92a3-64b7c4852a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" Oct 30 06:15:00.409750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.409711 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ba38f1-1102-4702-b073-f80c906b1363-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql\" (UID: \"67ba38f1-1102-4702-b073-f80c906b1363\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" Oct 30 06:15:00.410347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.410288 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af42bac-496e-4ffa-92a3-64b7c4852a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f\" (UID: \"6af42bac-496e-4ffa-92a3-64b7c4852a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" Oct 30 06:15:00.410347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.410317 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79946c52-915f-4a02-bce3-3d4c92c8360b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc\" (UID: \"79946c52-915f-4a02-bce3-3d4c92c8360b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" Oct 30 06:15:00.410347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.410289 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b87dd6d-e591-475a-80b4-389052f0c713-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q\" (UID: \"5b87dd6d-e591-475a-80b4-389052f0c713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" Oct 30 06:15:00.411019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.410992 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b4a966-5447-45de-ab4a-c79b929a3d78-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889015-x9tnr\" (UID: \"43b4a966-5447-45de-ab4a-c79b929a3d78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" Oct 30 06:15:00.439776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.439741 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wjj\" (UniqueName: \"kubernetes.io/projected/5b87dd6d-e591-475a-80b4-389052f0c713-kube-api-access-m7wjj\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q\" (UID: \"5b87dd6d-e591-475a-80b4-389052f0c713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" Oct 30 06:15:00.440034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.440008 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw674\" (UniqueName: \"kubernetes.io/projected/6af42bac-496e-4ffa-92a3-64b7c4852a73-kube-api-access-kw674\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f\" (UID: \"6af42bac-496e-4ffa-92a3-64b7c4852a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" Oct 30 06:15:00.440972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.440941 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w8zf\" (UniqueName: \"kubernetes.io/projected/43b4a966-5447-45de-ab4a-c79b929a3d78-kube-api-access-9w8zf\") pod \"maas-api-key-cleanup-29889015-x9tnr\" (UID: \"43b4a966-5447-45de-ab4a-c79b929a3d78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" Oct 30 06:15:00.441508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.441490 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfgvv\" (UniqueName: \"kubernetes.io/projected/67ba38f1-1102-4702-b073-f80c906b1363-kube-api-access-lfgvv\") pod \"maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql\" (UID: \"67ba38f1-1102-4702-b073-f80c906b1363\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" Oct 30 06:15:00.441703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.441683 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrzp\" (UniqueName: \"kubernetes.io/projected/79946c52-915f-4a02-bce3-3d4c92c8360b-kube-api-access-qzrzp\") pod \"maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc\" (UID: \"79946c52-915f-4a02-bce3-3d4c92c8360b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" Oct 30 06:15:00.511211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.511063 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9\" (UID: \"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" Oct 30 06:15:00.511211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.511179 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmgvw\" (UniqueName: \"kubernetes.io/projected/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-kube-api-access-dmgvw\") pod \"maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9\" (UID: \"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" Oct 30 06:15:00.511495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.511213 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fctxx\" (UniqueName: \"kubernetes.io/projected/592c1065-0e61-4095-8234-5cba81dd1283-kube-api-access-fctxx\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z\" (UID: \"592c1065-0e61-4095-8234-5cba81dd1283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" Oct 30 06:15:00.511495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.511274 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/592c1065-0e61-4095-8234-5cba81dd1283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z\" (UID: \"592c1065-0e61-4095-8234-5cba81dd1283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" Oct 30 06:15:00.512195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.512139 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/592c1065-0e61-4095-8234-5cba81dd1283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z\" (UID: \"592c1065-0e61-4095-8234-5cba81dd1283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" Oct 30 06:15:00.512772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.512750 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9\" (UID: \"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" Oct 30 06:15:00.518904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.518873 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" Oct 30 06:15:00.537903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.537869 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmgvw\" (UniqueName: \"kubernetes.io/projected/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-kube-api-access-dmgvw\") pod \"maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9\" (UID: \"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" Oct 30 06:15:00.538166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.538139 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fctxx\" (UniqueName: \"kubernetes.io/projected/592c1065-0e61-4095-8234-5cba81dd1283-kube-api-access-fctxx\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z\" (UID: \"592c1065-0e61-4095-8234-5cba81dd1283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" Oct 30 06:15:00.588817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.588778 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" Oct 30 06:15:00.611807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.605391 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" Oct 30 06:15:00.618055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.617860 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" Oct 30 06:15:00.635178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.634971 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" Oct 30 06:15:00.663123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.662919 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" Oct 30 06:15:00.667532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.667343 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" Oct 30 06:15:01.000364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.996577 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" event={"ID":"d3164cd0-0181-4645-9332-f82110a5e5c9","Type":"ContainerDied","Data":"0193716c5c69c6a8ec0083196b794006282f71e5544d30b5d6e43aaf145f66e3"} Oct 30 06:15:01.000364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.996637 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0193716c5c69c6a8ec0083196b794006282f71e5544d30b5d6e43aaf145f66e3" Oct 30 06:15:01.000364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:00.996713 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb" Oct 30 06:15:01.166328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:01.166279 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql"] Oct 30 06:15:01.172042 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:15:01.172014 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ba38f1_1102_4702_b073_f80c906b1363.slice/crio-d0200a3fe99ee69a7f114634e68364a197981c9f66716f09e0987855941cf312 WatchSource:0}: Error finding container d0200a3fe99ee69a7f114634e68364a197981c9f66716f09e0987855941cf312: Status 404 returned error can't find the container with id d0200a3fe99ee69a7f114634e68364a197981c9f66716f09e0987855941cf312 Oct 30 06:15:01.267212 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:15:01.267172 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af42bac_496e_4ffa_92a3_64b7c4852a73.slice/crio-9cd863705c74963703b975d4fafdb329caebf34b35f692611b5239916da19564 WatchSource:0}: Error finding container 9cd863705c74963703b975d4fafdb329caebf34b35f692611b5239916da19564: Status 404 returned error can't find the container with id 9cd863705c74963703b975d4fafdb329caebf34b35f692611b5239916da19564 Oct 30 06:15:01.278348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:01.278293 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f"] Oct 30 06:15:01.349446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:01.349420 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z"] Oct 30 06:15:01.351045 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:15:01.351006 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592c1065_0e61_4095_8234_5cba81dd1283.slice/crio-505660ae676b078c06625838fba2726c764da0f14b49804a7e7cf6e62fdea89f WatchSource:0}: Error finding container 505660ae676b078c06625838fba2726c764da0f14b49804a7e7cf6e62fdea89f: Status 404 returned error can't find the container with id 505660ae676b078c06625838fba2726c764da0f14b49804a7e7cf6e62fdea89f Oct 30 06:15:01.370689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:01.370653 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr"] Oct 30 06:15:01.377538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:01.377505 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q"] Oct 30 06:15:01.414459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:01.414411 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9"] Oct 30 06:15:01.427750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:01.427723 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc"] Oct 30 06:15:01.461254 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:15:01.461196 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b4a966_5447_45de_ab4a_c79b929a3d78.slice/crio-aa5e05271cb0cb4f72a763f9525ad48219eb0c292cdb2adddcf6ee06adf06b1b WatchSource:0}: Error finding container aa5e05271cb0cb4f72a763f9525ad48219eb0c292cdb2adddcf6ee06adf06b1b: Status 404 returned error can't find the container with id aa5e05271cb0cb4f72a763f9525ad48219eb0c292cdb2adddcf6ee06adf06b1b Oct 30 06:15:01.467456 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:15:01.467428 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5aba07_ee9a_4e79_bb19_a83ad37cec3b.slice/crio-e64c840322bccf28790eedad1e3e4c4c56f1fb5fca0c839292a0de32a0cf496c WatchSource:0}: Error finding container e64c840322bccf28790eedad1e3e4c4c56f1fb5fca0c839292a0de32a0cf496c: Status 404 returned error can't find the container with id e64c840322bccf28790eedad1e3e4c4c56f1fb5fca0c839292a0de32a0cf496c Oct 30 06:15:02.013052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.013015 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" event={"ID":"592c1065-0e61-4095-8234-5cba81dd1283","Type":"ContainerStarted","Data":"505660ae676b078c06625838fba2726c764da0f14b49804a7e7cf6e62fdea89f"} Oct 30 06:15:02.014994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.014960 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" event={"ID":"67ba38f1-1102-4702-b073-f80c906b1363","Type":"ContainerStarted","Data":"c9efefe0f9e4fd770213d33ef495495f1a3793e2172f3e6f1d0f37460388c9a7"} Oct 30 06:15:02.015139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.015025 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" event={"ID":"67ba38f1-1102-4702-b073-f80c906b1363","Type":"ContainerStarted","Data":"d0200a3fe99ee69a7f114634e68364a197981c9f66716f09e0987855941cf312"} Oct 30 06:15:02.017050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.017022 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" event={"ID":"5b87dd6d-e591-475a-80b4-389052f0c713","Type":"ContainerStarted","Data":"25cd564c45166664566d8fa6735cbdf60537fcc55977ebf0f43fabfdec463be4"} Oct 30 06:15:02.017164 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.017075 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" event={"ID":"5b87dd6d-e591-475a-80b4-389052f0c713","Type":"ContainerStarted","Data":"358be630feaa48aed8084955a5755b0aef8031a7364d84e8fc3bfd7911b89b75"} Oct 30 06:15:02.019096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.019073 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" event={"ID":"43b4a966-5447-45de-ab4a-c79b929a3d78","Type":"ContainerStarted","Data":"09b25e534af7f5d9940efb4dcd78730f7a55d9269ba261e4a80396b3283f4fc5"} Oct 30 06:15:02.019209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.019111 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" event={"ID":"43b4a966-5447-45de-ab4a-c79b929a3d78","Type":"ContainerStarted","Data":"aa5e05271cb0cb4f72a763f9525ad48219eb0c292cdb2adddcf6ee06adf06b1b"} Oct 30 06:15:02.021662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.021639 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" event={"ID":"6af42bac-496e-4ffa-92a3-64b7c4852a73","Type":"ContainerStarted","Data":"eb0742049e12d05e9f24b8ddca334b578f6deb2b9bc23cc30ad67e9e2190cd06"} Oct 30 06:15:02.021832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.021796 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" event={"ID":"6af42bac-496e-4ffa-92a3-64b7c4852a73","Type":"ContainerStarted","Data":"9cd863705c74963703b975d4fafdb329caebf34b35f692611b5239916da19564"} Oct 30 06:15:02.023643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.023620 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" event={"ID":"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b","Type":"ContainerStarted","Data":"77965050b8eb0e9a35fa5b1fbeee3b0090d85c61f1624a3d3de1a62ffb2da2c8"} Oct 30 06:15:02.023729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.023662 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" event={"ID":"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b","Type":"ContainerStarted","Data":"e64c840322bccf28790eedad1e3e4c4c56f1fb5fca0c839292a0de32a0cf496c"} Oct 30 06:15:02.025835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.025809 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" event={"ID":"79946c52-915f-4a02-bce3-3d4c92c8360b","Type":"ContainerStarted","Data":"0d95a9ec717b4e8270895bbaa0abe1e24abf432f06ce3ec709728066c2dc6290"} Oct 30 06:15:02.025942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.025854 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" event={"ID":"79946c52-915f-4a02-bce3-3d4c92c8360b","Type":"ContainerStarted","Data":"b1c1b4ee5632f9ede45074a1372dba1204f96dd48767a2792d7de4eb3db18773"} Oct 30 06:15:02.063461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.063413 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" podStartSLOduration=2.063396709 podStartE2EDuration="2.063396709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.056294549 +0000 UTC m=+1316.052251560" watchObservedRunningTime="2026-10-30 06:15:02.063396709 +0000 UTC m=+1316.059353719" Oct 30 06:15:02.100516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.100460 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" podStartSLOduration=2.100444812 podStartE2EDuration="2.100444812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.097928434 +0000 UTC m=+1316.093885445" watchObservedRunningTime="2026-10-30 06:15:02.100444812 +0000 UTC m=+1316.096401821" Oct 30 06:15:02.140248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.140192 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" podStartSLOduration=2.140176637 podStartE2EDuration="2.140176637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.13874116 +0000 UTC m=+1316.134698170" watchObservedRunningTime="2026-10-30 06:15:02.140176637 +0000 UTC m=+1316.136133646" Oct 30 06:15:02.186435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.186378 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" podStartSLOduration=2.18635898 podStartE2EDuration="2.18635898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.180617576 +0000 UTC m=+1316.176574589" watchObservedRunningTime="2026-10-30 06:15:02.18635898 +0000 UTC m=+1316.182315992" Oct 30 06:15:02.229003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.228895 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" podStartSLOduration=2.228871715 podStartE2EDuration="2.228871715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.222256164 +0000 UTC m=+1316.218213172" watchObservedRunningTime="2026-10-30 06:15:02.228871715 +0000 UTC m=+1316.224828726" Oct 30 06:15:02.267398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:02.267347 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" podStartSLOduration=2.267329734 podStartE2EDuration="2.267329734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.265344758 +0000 UTC m=+1316.261301769" watchObservedRunningTime="2026-10-30 06:15:02.267329734 +0000 UTC m=+1316.263286745" Oct 30 06:15:03.037211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.037124 5001 generic.go:356] "Generic (PLEG): container finished" podID="5b87dd6d-e591-475a-80b4-389052f0c713" containerID="25cd564c45166664566d8fa6735cbdf60537fcc55977ebf0f43fabfdec463be4" exitCode=60 Oct 30 06:15:03.037482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.037205 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" event={"ID":"5b87dd6d-e591-475a-80b4-389052f0c713","Type":"ContainerDied","Data":"25cd564c45166664566d8fa6735cbdf60537fcc55977ebf0f43fabfdec463be4"} Oct 30 06:15:03.037553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.037535 5001 scope.go:122] "RemoveContainer" containerID="25cd564c45166664566d8fa6735cbdf60537fcc55977ebf0f43fabfdec463be4" Oct 30 06:15:03.039112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.039088 5001 generic.go:356] "Generic (PLEG): container finished" podID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerID="09b25e534af7f5d9940efb4dcd78730f7a55d9269ba261e4a80396b3283f4fc5" exitCode=60 Oct 30 06:15:03.039203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.039161 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" event={"ID":"43b4a966-5447-45de-ab4a-c79b929a3d78","Type":"ContainerDied","Data":"09b25e534af7f5d9940efb4dcd78730f7a55d9269ba261e4a80396b3283f4fc5"} Oct 30 06:15:03.039454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.039433 5001 scope.go:122] "RemoveContainer" containerID="09b25e534af7f5d9940efb4dcd78730f7a55d9269ba261e4a80396b3283f4fc5" Oct 30 06:15:03.041388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.041366 5001 generic.go:356] "Generic (PLEG): container finished" podID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerID="eb0742049e12d05e9f24b8ddca334b578f6deb2b9bc23cc30ad67e9e2190cd06" exitCode=60 Oct 30 06:15:03.041545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.041389 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" event={"ID":"6af42bac-496e-4ffa-92a3-64b7c4852a73","Type":"ContainerDied","Data":"eb0742049e12d05e9f24b8ddca334b578f6deb2b9bc23cc30ad67e9e2190cd06"} Oct 30 06:15:03.041717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.041700 5001 scope.go:122] "RemoveContainer" containerID="eb0742049e12d05e9f24b8ddca334b578f6deb2b9bc23cc30ad67e9e2190cd06" Oct 30 06:15:03.043299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.043283 5001 generic.go:356] "Generic (PLEG): container finished" podID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerID="77965050b8eb0e9a35fa5b1fbeee3b0090d85c61f1624a3d3de1a62ffb2da2c8" exitCode=60 Oct 30 06:15:03.043399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.043367 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" event={"ID":"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b","Type":"ContainerDied","Data":"77965050b8eb0e9a35fa5b1fbeee3b0090d85c61f1624a3d3de1a62ffb2da2c8"} Oct 30 06:15:03.043609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.043595 5001 scope.go:122] "RemoveContainer" containerID="77965050b8eb0e9a35fa5b1fbeee3b0090d85c61f1624a3d3de1a62ffb2da2c8" Oct 30 06:15:03.045245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.045221 5001 generic.go:356] "Generic (PLEG): container finished" podID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerID="1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253" exitCode=0 Oct 30 06:15:03.045360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.045283 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" event={"ID":"8cfbab7a-23c7-4712-b0c6-65529f21890e","Type":"ContainerDied","Data":"1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253"} Oct 30 06:15:03.045958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.045935 5001 scope.go:122] "RemoveContainer" containerID="1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253" Oct 30 06:15:03.047393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.047374 5001 generic.go:356] "Generic (PLEG): container finished" podID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerID="0d95a9ec717b4e8270895bbaa0abe1e24abf432f06ce3ec709728066c2dc6290" exitCode=60 Oct 30 06:15:03.047493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.047445 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" event={"ID":"79946c52-915f-4a02-bce3-3d4c92c8360b","Type":"ContainerDied","Data":"0d95a9ec717b4e8270895bbaa0abe1e24abf432f06ce3ec709728066c2dc6290"} Oct 30 06:15:03.047676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.047654 5001 scope.go:122] "RemoveContainer" containerID="0d95a9ec717b4e8270895bbaa0abe1e24abf432f06ce3ec709728066c2dc6290" Oct 30 06:15:03.049590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.049563 5001 generic.go:356] "Generic (PLEG): container finished" podID="592c1065-0e61-4095-8234-5cba81dd1283" containerID="11db885d8260d36956f2dc34a5f1793275a9083ece124580ef47d4e57d578bd8" exitCode=60 Oct 30 06:15:03.049708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.049635 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" event={"ID":"592c1065-0e61-4095-8234-5cba81dd1283","Type":"ContainerDied","Data":"11db885d8260d36956f2dc34a5f1793275a9083ece124580ef47d4e57d578bd8"} Oct 30 06:15:03.049917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.049899 5001 scope.go:122] "RemoveContainer" containerID="11db885d8260d36956f2dc34a5f1793275a9083ece124580ef47d4e57d578bd8" Oct 30 06:15:03.054330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.053350 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:03.054330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.053567 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:03.054330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.053692 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:03.072726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.072410 5001 generic.go:356] "Generic (PLEG): container finished" podID="67ba38f1-1102-4702-b073-f80c906b1363" containerID="c9efefe0f9e4fd770213d33ef495495f1a3793e2172f3e6f1d0f37460388c9a7" exitCode=7 Oct 30 06:15:03.072726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.072625 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" event={"ID":"67ba38f1-1102-4702-b073-f80c906b1363","Type":"ContainerDied","Data":"c9efefe0f9e4fd770213d33ef495495f1a3793e2172f3e6f1d0f37460388c9a7"} Oct 30 06:15:03.074287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.074265 5001 scope.go:122] "RemoveContainer" containerID="c9efefe0f9e4fd770213d33ef495495f1a3793e2172f3e6f1d0f37460388c9a7" Oct 30 06:15:03.908395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:03.907975 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:15:04.103805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.103760 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" event={"ID":"43b4a966-5447-45de-ab4a-c79b929a3d78","Type":"ContainerStarted","Data":"3b744d7f72fff3fae59c4d5900a8564a3533378559d8dd9f106bb949b31372b1"} Oct 30 06:15:04.108192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.108160 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" event={"ID":"6af42bac-496e-4ffa-92a3-64b7c4852a73","Type":"ContainerStarted","Data":"81fee31854172eea5086ca4aa547dc4213503e51ee5f6d2c04ad13d992289a08"} Oct 30 06:15:04.237384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.233406 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" event={"ID":"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b","Type":"ContainerStarted","Data":"0c8f1d87634c6598f405778416ebe62bbd84bf7de29ef48d7a6506cf348a89cc"} Oct 30 06:15:04.237733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.237698 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" event={"ID":"8cfbab7a-23c7-4712-b0c6-65529f21890e","Type":"ContainerStarted","Data":"a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964"} Oct 30 06:15:04.294610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.277965 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" event={"ID":"79946c52-915f-4a02-bce3-3d4c92c8360b","Type":"ContainerStarted","Data":"001c4459a205d4f38613f6e3e6d447f9c6e8934fe1c62568994d1e740e2d5cbf"} Oct 30 06:15:04.347377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.340443 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" event={"ID":"592c1065-0e61-4095-8234-5cba81dd1283","Type":"ContainerStarted","Data":"3d200f08b540fa64f34a6f2ba33d70e602711c253d0e1c5a6f6e46b230c18ecf"} Oct 30 06:15:04.373070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.373029 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" event={"ID":"67ba38f1-1102-4702-b073-f80c906b1363","Type":"ContainerStarted","Data":"32685e481278b4704155361d734e72a60b13d26ee5258732babc72e2ec48759f"} Oct 30 06:15:04.381908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.381876 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" event={"ID":"5b87dd6d-e591-475a-80b4-389052f0c713","Type":"ContainerStarted","Data":"2c732532d51598b8991786f29e159bd947b57d4c8b6143f5e6ce756227175271"} Oct 30 06:15:04.485439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.463866 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:15:04.503384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.488490 5001 patch_prober.go:29] interesting pod/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:15:04.503384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.488569 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 06:15:04.812057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.811940 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" podStartSLOduration=4.811903263 podStartE2EDuration="4.811903263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.548181836 +0000 UTC m=+1318.544138858" watchObservedRunningTime="2026-10-30 06:15:04.811903263 +0000 UTC m=+1318.807860274" Oct 30 06:15:04.915523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:04.915493 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:15:05.395586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.395549 5001 generic.go:356] "Generic (PLEG): container finished" podID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerID="3b744d7f72fff3fae59c4d5900a8564a3533378559d8dd9f106bb949b31372b1" exitCode=60 Oct 30 06:15:05.395807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.395626 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" event={"ID":"43b4a966-5447-45de-ab4a-c79b929a3d78","Type":"ContainerDied","Data":"3b744d7f72fff3fae59c4d5900a8564a3533378559d8dd9f106bb949b31372b1"} Oct 30 06:15:05.395807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.395701 5001 scope.go:122] "RemoveContainer" containerID="09b25e534af7f5d9940efb4dcd78730f7a55d9269ba261e4a80396b3283f4fc5" Oct 30 06:15:05.396256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.396119 5001 scope.go:122] "RemoveContainer" containerID="3b744d7f72fff3fae59c4d5900a8564a3533378559d8dd9f106bb949b31372b1" Oct 30 06:15:05.397420 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:05.396589 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889015-x9tnr_odh-ai-gateway-infra(43b4a966-5447-45de-ab4a-c79b929a3d78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" Oct 30 06:15:05.398769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.398741 5001 generic.go:356] "Generic (PLEG): container finished" podID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerID="81fee31854172eea5086ca4aa547dc4213503e51ee5f6d2c04ad13d992289a08" exitCode=60 Oct 30 06:15:05.398892 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.398771 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" event={"ID":"6af42bac-496e-4ffa-92a3-64b7c4852a73","Type":"ContainerDied","Data":"81fee31854172eea5086ca4aa547dc4213503e51ee5f6d2c04ad13d992289a08"} Oct 30 06:15:05.399181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.399166 5001 scope.go:122] "RemoveContainer" containerID="81fee31854172eea5086ca4aa547dc4213503e51ee5f6d2c04ad13d992289a08" Oct 30 06:15:05.399739 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:05.399714 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f_odh-ai-gateway-infra(6af42bac-496e-4ffa-92a3-64b7c4852a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" Oct 30 06:15:05.401530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.401510 5001 generic.go:356] "Generic (PLEG): container finished" podID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerID="0c8f1d87634c6598f405778416ebe62bbd84bf7de29ef48d7a6506cf348a89cc" exitCode=60 Oct 30 06:15:05.401787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.401542 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" event={"ID":"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b","Type":"ContainerDied","Data":"0c8f1d87634c6598f405778416ebe62bbd84bf7de29ef48d7a6506cf348a89cc"} Oct 30 06:15:05.402037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.401854 5001 scope.go:122] "RemoveContainer" containerID="0c8f1d87634c6598f405778416ebe62bbd84bf7de29ef48d7a6506cf348a89cc" Oct 30 06:15:05.402582 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:05.402560 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9_odh-ai-gateway-infra(aa5aba07-ee9a-4e79-bb19-a83ad37cec3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" Oct 30 06:15:05.404627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.404603 5001 generic.go:356] "Generic (PLEG): container finished" podID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerID="001c4459a205d4f38613f6e3e6d447f9c6e8934fe1c62568994d1e740e2d5cbf" exitCode=60 Oct 30 06:15:05.404888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.404858 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" event={"ID":"79946c52-915f-4a02-bce3-3d4c92c8360b","Type":"ContainerDied","Data":"001c4459a205d4f38613f6e3e6d447f9c6e8934fe1c62568994d1e740e2d5cbf"} Oct 30 06:15:05.405388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.405368 5001 scope.go:122] "RemoveContainer" containerID="001c4459a205d4f38613f6e3e6d447f9c6e8934fe1c62568994d1e740e2d5cbf" Oct 30 06:15:05.408584 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:05.408556 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc_odh-ai-gateway-infra(79946c52-915f-4a02-bce3-3d4c92c8360b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" Oct 30 06:15:05.409023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.408998 5001 generic.go:356] "Generic (PLEG): container finished" podID="592c1065-0e61-4095-8234-5cba81dd1283" containerID="3d200f08b540fa64f34a6f2ba33d70e602711c253d0e1c5a6f6e46b230c18ecf" exitCode=60 Oct 30 06:15:05.409141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.409071 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" event={"ID":"592c1065-0e61-4095-8234-5cba81dd1283","Type":"ContainerDied","Data":"3d200f08b540fa64f34a6f2ba33d70e602711c253d0e1c5a6f6e46b230c18ecf"} Oct 30 06:15:05.409395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.409378 5001 scope.go:122] "RemoveContainer" containerID="3d200f08b540fa64f34a6f2ba33d70e602711c253d0e1c5a6f6e46b230c18ecf" Oct 30 06:15:05.409734 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:05.409705 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z_odh-ai-gateway-infra(592c1065-0e61-4095-8234-5cba81dd1283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" podUID="592c1065-0e61-4095-8234-5cba81dd1283" Oct 30 06:15:05.411370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.411345 5001 generic.go:356] "Generic (PLEG): container finished" podID="67ba38f1-1102-4702-b073-f80c906b1363" containerID="32685e481278b4704155361d734e72a60b13d26ee5258732babc72e2ec48759f" exitCode=7 Oct 30 06:15:05.411473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.411398 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" event={"ID":"67ba38f1-1102-4702-b073-f80c906b1363","Type":"ContainerDied","Data":"32685e481278b4704155361d734e72a60b13d26ee5258732babc72e2ec48759f"} Oct 30 06:15:05.411765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.411748 5001 scope.go:122] "RemoveContainer" containerID="32685e481278b4704155361d734e72a60b13d26ee5258732babc72e2ec48759f" Oct 30 06:15:05.412092 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:05.412069 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql_odh-ai-gateway-infra(67ba38f1-1102-4702-b073-f80c906b1363)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" podUID="67ba38f1-1102-4702-b073-f80c906b1363" Oct 30 06:15:05.413782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.413763 5001 generic.go:356] "Generic (PLEG): container finished" podID="5b87dd6d-e591-475a-80b4-389052f0c713" containerID="2c732532d51598b8991786f29e159bd947b57d4c8b6143f5e6ce756227175271" exitCode=60 Oct 30 06:15:05.413892 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.413838 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" event={"ID":"5b87dd6d-e591-475a-80b4-389052f0c713","Type":"ContainerDied","Data":"2c732532d51598b8991786f29e159bd947b57d4c8b6143f5e6ce756227175271"} Oct 30 06:15:05.414092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.414075 5001 scope.go:122] "RemoveContainer" containerID="2c732532d51598b8991786f29e159bd947b57d4c8b6143f5e6ce756227175271" Oct 30 06:15:05.414443 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:05.414417 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q_odh-ai-gateway-infra(5b87dd6d-e591-475a-80b4-389052f0c713)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" Oct 30 06:15:05.414513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.414495 5001 scope.go:122] "RemoveContainer" containerID="eb0742049e12d05e9f24b8ddca334b578f6deb2b9bc23cc30ad67e9e2190cd06" Oct 30 06:15:05.428753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.428736 5001 scope.go:122] "RemoveContainer" containerID="77965050b8eb0e9a35fa5b1fbeee3b0090d85c61f1624a3d3de1a62ffb2da2c8" Oct 30 06:15:05.441690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.441668 5001 scope.go:122] "RemoveContainer" containerID="0d95a9ec717b4e8270895bbaa0abe1e24abf432f06ce3ec709728066c2dc6290" Oct 30 06:15:05.473340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.463061 5001 patch_prober.go:29] interesting pod/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:15:05.473340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.463125 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 06:15:05.492532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.492509 5001 scope.go:122] "RemoveContainer" containerID="11db885d8260d36956f2dc34a5f1793275a9083ece124580ef47d4e57d578bd8" Oct 30 06:15:05.506868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.506843 5001 scope.go:122] "RemoveContainer" containerID="c9efefe0f9e4fd770213d33ef495495f1a3793e2172f3e6f1d0f37460388c9a7" Oct 30 06:15:05.523297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:05.523271 5001 scope.go:122] "RemoveContainer" containerID="25cd564c45166664566d8fa6735cbdf60537fcc55977ebf0f43fabfdec463be4" Oct 30 06:15:06.424754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.424724 5001 scope.go:122] "RemoveContainer" containerID="3d200f08b540fa64f34a6f2ba33d70e602711c253d0e1c5a6f6e46b230c18ecf" Oct 30 06:15:06.425185 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:06.425139 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z_odh-ai-gateway-infra(592c1065-0e61-4095-8234-5cba81dd1283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" podUID="592c1065-0e61-4095-8234-5cba81dd1283" Oct 30 06:15:06.426528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.426497 5001 scope.go:122] "RemoveContainer" containerID="32685e481278b4704155361d734e72a60b13d26ee5258732babc72e2ec48759f" Oct 30 06:15:06.426866 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:06.426840 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql_odh-ai-gateway-infra(67ba38f1-1102-4702-b073-f80c906b1363)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" podUID="67ba38f1-1102-4702-b073-f80c906b1363" Oct 30 06:15:06.428253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.428224 5001 scope.go:122] "RemoveContainer" containerID="2c732532d51598b8991786f29e159bd947b57d4c8b6143f5e6ce756227175271" Oct 30 06:15:06.428568 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:06.428543 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q_odh-ai-gateway-infra(5b87dd6d-e591-475a-80b4-389052f0c713)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" Oct 30 06:15:06.429871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.429855 5001 scope.go:122] "RemoveContainer" containerID="3b744d7f72fff3fae59c4d5900a8564a3533378559d8dd9f106bb949b31372b1" Oct 30 06:15:06.430116 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:06.430096 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889015-x9tnr_odh-ai-gateway-infra(43b4a966-5447-45de-ab4a-c79b929a3d78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" Oct 30 06:15:06.431466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.431448 5001 scope.go:122] "RemoveContainer" containerID="81fee31854172eea5086ca4aa547dc4213503e51ee5f6d2c04ad13d992289a08" Oct 30 06:15:06.431703 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:06.431682 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f_odh-ai-gateway-infra(6af42bac-496e-4ffa-92a3-64b7c4852a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" Oct 30 06:15:06.432942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.432922 5001 scope.go:122] "RemoveContainer" containerID="0c8f1d87634c6598f405778416ebe62bbd84bf7de29ef48d7a6506cf348a89cc" Oct 30 06:15:06.433159 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:06.433142 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9_odh-ai-gateway-infra(aa5aba07-ee9a-4e79-bb19-a83ad37cec3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" Oct 30 06:15:06.434604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.434588 5001 scope.go:122] "RemoveContainer" containerID="001c4459a205d4f38613f6e3e6d447f9c6e8934fe1c62568994d1e740e2d5cbf" Oct 30 06:15:06.434771 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:06.434755 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc_odh-ai-gateway-infra(79946c52-915f-4a02-bce3-3d4c92c8360b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" Oct 30 06:15:06.465332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.460712 5001 patch_prober.go:29] interesting pod/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:15:06.465332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.460783 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 06:15:06.850451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.841902 5001 scope.go:122] "RemoveContainer" containerID="7fee3710cbf570491fdc240b3198b9cb6ba7b618607ffe9136c01203ddfb12de" Oct 30 06:15:06.869629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.863278 5001 scope.go:122] "RemoveContainer" containerID="bfc0d53448bd9d2e7f4e11c85bbfd3fdf338c2069f98aca3af882d5b2bba5d17" Oct 30 06:15:06.902553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.902529 5001 scope.go:122] "RemoveContainer" containerID="d2bdf61371c11db882a7f94258a23d403dfd2405eb98ed4637cb9312ecb0b11b" Oct 30 06:15:06.916121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:06.916086 5001 scope.go:122] "RemoveContainer" containerID="c14e3cbed58eb7f574e65eff280648ffc7cffeb55abe776043a3f8f320824754" Oct 30 06:15:07.190199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:07.190158 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx"] Oct 30 06:15:07.473398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:07.472178 5001 patch_prober.go:29] interesting pod/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:15:07.473398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:07.472239 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 06:15:08.300858 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:08.300826 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:15:08.468138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:08.467678 5001 patch_prober.go:29] interesting pod/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:15:08.468138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:08.467737 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 06:15:09.465207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:09.462665 5001 patch_prober.go:29] interesting pod/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:15:09.465207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:09.462738 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 06:15:10.528659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:10.524257 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:15:10.528659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:10.526000 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:15:10.528659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:10.527083 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:15:10.973371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:10.957678 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf"] Oct 30 06:15:11.094200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.094156 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f789d7-qffpf"] Oct 30 06:15:11.468725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.468689 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2"] Oct 30 06:15:11.507742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.498019 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.515695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.515664 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f7a606-openshift-default-dockercfg-tnx85\"" Oct 30 06:15:11.516947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.516923 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2"] Oct 30 06:15:11.627485 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.627451 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-envoy\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.627875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.627494 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-socket\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.627875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.627558 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-token\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.627875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.627694 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istiod-ca-cert\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.628010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.627879 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-podinfo\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.628010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.627986 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggnqp\" (UniqueName: \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-kube-api-access-ggnqp\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.628077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.628046 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-certs\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.628174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.628155 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-credential-socket\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.628240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.628225 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-data\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.731228 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggnqp\" (UniqueName: \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-kube-api-access-ggnqp\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.731288 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-certs\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.731445 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-credential-socket\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.731522 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-data\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.731610 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-envoy\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.731648 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-socket\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.731681 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-token\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.731776 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istiod-ca-cert\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.731943 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-podinfo\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.734549 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-certs\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.735263 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-credential-socket\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.735663 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-socket\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.739384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.735917 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-data\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.750350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.749460 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istiod-ca-cert\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.782329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.775597 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-podinfo\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.798340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.791699 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-envoy\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.821336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.820634 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggnqp\" (UniqueName: \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-kube-api-access-ggnqp\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:11.823539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:11.823482 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-token\") pod \"e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:12.118734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.118700 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a62a74-8357-4261-9f6c-2da2f388d02f" path="/var/lib/kubelet/pods/68a62a74-8357-4261-9f6c-2da2f388d02f/volumes" Oct 30 06:15:12.121261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.121241 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:12.190667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.190606 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" podUID="dd3c9884-90ba-4058-a58b-da51bb4008ea" containerName="payload-pre-processing" containerID="cri-o://566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624" gracePeriod=25 Oct 30 06:15:12.457975 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:15:12.457744 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6842a8_36df_4ea9_8639_a7a7cb18e577.slice/crio-df21455ca848e33cdf150b81bbca79f01169d1863457f6b5474544e7328277c8 WatchSource:0}: Error finding container df21455ca848e33cdf150b81bbca79f01169d1863457f6b5474544e7328277c8: Status 404 returned error can't find the container with id df21455ca848e33cdf150b81bbca79f01169d1863457f6b5474544e7328277c8 Oct 30 06:15:12.462620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.461897 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:12.462620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.462009 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:12.462620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.462051 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:12.468951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.468777 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2"] Oct 30 06:15:12.508353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.506650 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:15:12.542939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.542846 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3c9884-90ba-4058-a58b-da51bb4008ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3c9884-90ba-4058-a58b-da51bb4008ea-plugins-config-volume\") pod \"dd3c9884-90ba-4058-a58b-da51bb4008ea\" (UID: \"dd3c9884-90ba-4058-a58b-da51bb4008ea\") " Oct 30 06:15:12.542939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.542968 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3c9884-90ba-4058-a58b-da51bb4008ea-kube-api-access-sgplp\" (UniqueName: \"kubernetes.io/projected/dd3c9884-90ba-4058-a58b-da51bb4008ea-kube-api-access-sgplp\") pod \"dd3c9884-90ba-4058-a58b-da51bb4008ea\" (UID: \"dd3c9884-90ba-4058-a58b-da51bb4008ea\") " Oct 30 06:15:12.544143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.544077 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3c9884-90ba-4058-a58b-da51bb4008ea-plugins-config-volume" pod "dd3c9884-90ba-4058-a58b-da51bb4008ea" (UID: "dd3c9884-90ba-4058-a58b-da51bb4008ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:12.546369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.546318 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3c9884-90ba-4058-a58b-da51bb4008ea-kube-api-access-sgplp" pod "dd3c9884-90ba-4058-a58b-da51bb4008ea" (UID: "dd3c9884-90ba-4058-a58b-da51bb4008ea"). InnerVolumeSpecName "kube-api-access-sgplp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:12.645060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.644962 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgplp\" (UniqueName: \"kubernetes.io/projected/dd3c9884-90ba-4058-a58b-da51bb4008ea-kube-api-access-sgplp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:12.645060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.645001 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3c9884-90ba-4058-a58b-da51bb4008ea-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:12.680690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.678057 5001 generic.go:356] "Generic (PLEG): container finished" podID="dd3c9884-90ba-4058-a58b-da51bb4008ea" containerID="566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624" exitCode=0 Oct 30 06:15:12.680690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.678233 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" event={"ID":"dd3c9884-90ba-4058-a58b-da51bb4008ea","Type":"ContainerDied","Data":"566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624"} Oct 30 06:15:12.680690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.678284 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" event={"ID":"dd3c9884-90ba-4058-a58b-da51bb4008ea","Type":"ContainerDied","Data":"216cb7cce32f30eb1a25087b6ca2d6317019ce5f0b80127e0853f8f7e4456a79"} Oct 30 06:15:12.680690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.678441 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx" Oct 30 06:15:12.683101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.683077 5001 scope.go:122] "RemoveContainer" containerID="566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624" Oct 30 06:15:12.686474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.686443 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" event={"ID":"bf6842a8-36df-4ea9-8639-a7a7cb18e577","Type":"ContainerStarted","Data":"db541d13c4fdd3522aec928d0985373f7374bf475fa416591adb849b12897636"} Oct 30 06:15:12.686647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.686632 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" event={"ID":"bf6842a8-36df-4ea9-8639-a7a7cb18e577","Type":"ContainerStarted","Data":"df21455ca848e33cdf150b81bbca79f01169d1863457f6b5474544e7328277c8"} Oct 30 06:15:12.709378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.709352 5001 scope.go:122] "RemoveContainer" containerID="566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624" Oct 30 06:15:12.709740 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:12.709718 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624\": container with ID starting with 566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624 not found: ID does not exist" containerID="566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624" Oct 30 06:15:12.709822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.709749 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624"} err="failed to get container status \"566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624\": rpc error: code = NotFound desc = could not find container \"566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624\": container with ID starting with 566623cec7b4c778883a00ba38a6cc8e3cd080abb3ae1468dc0984321869a624 not found: ID does not exist" Oct 30 06:15:12.744929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.744891 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx"] Oct 30 06:15:12.751011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.750975 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e6df62-fd6b46799-ks5zx"] Oct 30 06:15:12.796099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:12.796035 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" podStartSLOduration=1.7960165529999998 podStartE2EDuration="1.796016553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:12.789240589 +0000 UTC m=+1326.785197600" watchObservedRunningTime="2026-10-30 06:15:12.796016553 +0000 UTC m=+1326.791973564" Oct 30 06:15:13.124328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:13.121815 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:13.129566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:13.129005 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:13.699890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:13.699856 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:13.700882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:13.700866 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:15:14.119567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:14.119528 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3c9884-90ba-4058-a58b-da51bb4008ea" path="/var/lib/kubelet/pods/dd3c9884-90ba-4058-a58b-da51bb4008ea/volumes" Oct 30 06:15:14.384385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:14.382810 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:15:17.110554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.110511 5001 scope.go:122] "RemoveContainer" containerID="0c8f1d87634c6598f405778416ebe62bbd84bf7de29ef48d7a6506cf348a89cc" Oct 30 06:15:17.111057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.110966 5001 scope.go:122] "RemoveContainer" containerID="2c732532d51598b8991786f29e159bd947b57d4c8b6143f5e6ce756227175271" Oct 30 06:15:17.111178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.111157 5001 scope.go:122] "RemoveContainer" containerID="3b744d7f72fff3fae59c4d5900a8564a3533378559d8dd9f106bb949b31372b1" Oct 30 06:15:17.450789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.448067 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6"] Oct 30 06:15:17.450789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.448863 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dd3c9884-90ba-4058-a58b-da51bb4008ea" containerName="payload-pre-processing" Oct 30 06:15:17.450789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.448877 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3c9884-90ba-4058-a58b-da51bb4008ea" containerName="payload-pre-processing" Oct 30 06:15:17.450789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.449570 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3c9884-90ba-4058-a58b-da51bb4008ea" containerName="payload-pre-processing" Oct 30 06:15:17.468297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.455192 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:15:17.502960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.494439 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f7a606-dockercfg-jvbmd\"" Oct 30 06:15:17.514956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.511783 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6"] Oct 30 06:15:17.542665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.542631 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f7a606\"" Oct 30 06:15:17.580326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.578799 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6\" (UID: \"2138fd05-a058-4a27-8b6f-4c43b1f5de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:15:17.580326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.578935 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqb46\" (UniqueName: \"kubernetes.io/projected/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-kube-api-access-qqb46\") pod \"payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6\" (UID: \"2138fd05-a058-4a27-8b6f-4c43b1f5de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:15:17.687335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.686496 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6\" (UID: \"2138fd05-a058-4a27-8b6f-4c43b1f5de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:15:17.687335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.686581 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqb46\" (UniqueName: \"kubernetes.io/projected/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-kube-api-access-qqb46\") pod \"payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6\" (UID: \"2138fd05-a058-4a27-8b6f-4c43b1f5de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:15:17.713423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:17.711051 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6\" (UID: \"2138fd05-a058-4a27-8b6f-4c43b1f5de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:15:18.104641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:18.089777 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqb46\" (UniqueName: \"kubernetes.io/projected/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-kube-api-access-qqb46\") pod \"payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6\" (UID: \"2138fd05-a058-4a27-8b6f-4c43b1f5de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:15:18.110266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:18.110214 5001 scope.go:122] "RemoveContainer" containerID="32685e481278b4704155361d734e72a60b13d26ee5258732babc72e2ec48759f" Oct 30 06:15:18.113137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:18.111771 5001 scope.go:122] "RemoveContainer" containerID="001c4459a205d4f38613f6e3e6d447f9c6e8934fe1c62568994d1e740e2d5cbf" Oct 30 06:15:18.120538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:18.120522 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:15:18.122889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:18.122865 5001 scope.go:122] "RemoveContainer" containerID="81fee31854172eea5086ca4aa547dc4213503e51ee5f6d2c04ad13d992289a08" Oct 30 06:15:18.586728 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:15:18.586696 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2138fd05_a058_4a27_8b6f_4c43b1f5de9c.slice/crio-02d688bdc8f9f0ad8562016a4576de3ddae697c0769afd5aec7f7d25aa66f40b WatchSource:0}: Error finding container 02d688bdc8f9f0ad8562016a4576de3ddae697c0769afd5aec7f7d25aa66f40b: Status 404 returned error can't find the container with id 02d688bdc8f9f0ad8562016a4576de3ddae697c0769afd5aec7f7d25aa66f40b Oct 30 06:15:18.592185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:18.591879 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6"] Oct 30 06:15:19.110318 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.110263 5001 scope.go:122] "RemoveContainer" containerID="3d200f08b540fa64f34a6f2ba33d70e602711c253d0e1c5a6f6e46b230c18ecf" Oct 30 06:15:19.122112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.121242 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" event={"ID":"79946c52-915f-4a02-bce3-3d4c92c8360b","Type":"ContainerStarted","Data":"03b2bb8c2063e1f7bffb7f0210357bcf50bfb2b46f00fd3940fbd1274eab9202"} Oct 30 06:15:19.126327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.126289 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" event={"ID":"67ba38f1-1102-4702-b073-f80c906b1363","Type":"ContainerStarted","Data":"bf955e2c85b6107bb532ad1bc2900bcbefd3017f31dc12905665b28800b615a1"} Oct 30 06:15:19.128573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.128550 5001 generic.go:356] "Generic (PLEG): container finished" podID="5b87dd6d-e591-475a-80b4-389052f0c713" containerID="2655e181d8097226e09b89064938967a55d6472d3e7173aa464ef8d653e2f7fe" exitCode=60 Oct 30 06:15:19.128814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.128667 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" event={"ID":"5b87dd6d-e591-475a-80b4-389052f0c713","Type":"ContainerDied","Data":"2655e181d8097226e09b89064938967a55d6472d3e7173aa464ef8d653e2f7fe"} Oct 30 06:15:19.128814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.128791 5001 scope.go:122] "RemoveContainer" containerID="2c732532d51598b8991786f29e159bd947b57d4c8b6143f5e6ce756227175271" Oct 30 06:15:19.129172 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.129156 5001 scope.go:122] "RemoveContainer" containerID="2655e181d8097226e09b89064938967a55d6472d3e7173aa464ef8d653e2f7fe" Oct 30 06:15:19.129442 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:19.129418 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q_odh-ai-gateway-infra(5b87dd6d-e591-475a-80b4-389052f0c713)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" Oct 30 06:15:19.131974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.131132 5001 generic.go:356] "Generic (PLEG): container finished" podID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerID="acbc097710183e8832cdc7c5131f598e7e8cadc664a798d86f9b51b42b06f13c" exitCode=60 Oct 30 06:15:19.131974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.131181 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" event={"ID":"43b4a966-5447-45de-ab4a-c79b929a3d78","Type":"ContainerDied","Data":"acbc097710183e8832cdc7c5131f598e7e8cadc664a798d86f9b51b42b06f13c"} Oct 30 06:15:19.131974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.131513 5001 scope.go:122] "RemoveContainer" containerID="acbc097710183e8832cdc7c5131f598e7e8cadc664a798d86f9b51b42b06f13c" Oct 30 06:15:19.131974 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:19.131723 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889015-x9tnr_odh-ai-gateway-infra(43b4a966-5447-45de-ab4a-c79b929a3d78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" Oct 30 06:15:19.134814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.134793 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" event={"ID":"6af42bac-496e-4ffa-92a3-64b7c4852a73","Type":"ContainerStarted","Data":"ff6c95e0d759a1052d5f032debc0eba2167c681c29853717a5ee609da3b4a240"} Oct 30 06:15:19.141294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.140758 5001 generic.go:356] "Generic (PLEG): container finished" podID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerID="dcf05a3add189c886a5ac855e5daf1fc3b39df0531e8b999b6606bbdb3e695c8" exitCode=60 Oct 30 06:15:19.141294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.140810 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" event={"ID":"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b","Type":"ContainerDied","Data":"dcf05a3add189c886a5ac855e5daf1fc3b39df0531e8b999b6606bbdb3e695c8"} Oct 30 06:15:19.141294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.141052 5001 scope.go:122] "RemoveContainer" containerID="dcf05a3add189c886a5ac855e5daf1fc3b39df0531e8b999b6606bbdb3e695c8" Oct 30 06:15:19.141294 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:19.141241 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9_odh-ai-gateway-infra(aa5aba07-ee9a-4e79-bb19-a83ad37cec3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" Oct 30 06:15:19.144048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.143665 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" event={"ID":"2138fd05-a058-4a27-8b6f-4c43b1f5de9c","Type":"ContainerStarted","Data":"0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093"} Oct 30 06:15:19.144048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.143700 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" event={"ID":"2138fd05-a058-4a27-8b6f-4c43b1f5de9c","Type":"ContainerStarted","Data":"02d688bdc8f9f0ad8562016a4576de3ddae697c0769afd5aec7f7d25aa66f40b"} Oct 30 06:15:19.144048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.143767 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:15:19.190939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.190904 5001 scope.go:122] "RemoveContainer" containerID="3b744d7f72fff3fae59c4d5900a8564a3533378559d8dd9f106bb949b31372b1" Oct 30 06:15:19.205760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.205732 5001 scope.go:122] "RemoveContainer" containerID="0c8f1d87634c6598f405778416ebe62bbd84bf7de29ef48d7a6506cf348a89cc" Oct 30 06:15:19.337300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:19.337246 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" podStartSLOduration=2.337233481 podStartE2EDuration="2.337233481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:19.292024364 +0000 UTC m=+1333.287981376" watchObservedRunningTime="2026-10-30 06:15:19.337233481 +0000 UTC m=+1333.333190491" Oct 30 06:15:20.210364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.206770 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc"] Oct 30 06:15:20.210867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.210464 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" event={"ID":"592c1065-0e61-4095-8234-5cba81dd1283","Type":"ContainerStarted","Data":"881ccf274fa07218cf3d509c53d25e5e721f90fca54745d74e6ab8c702dedfdf"} Oct 30 06:15:20.210935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.210873 5001 generic.go:356] "Generic (PLEG): container finished" podID="67ba38f1-1102-4702-b073-f80c906b1363" containerID="bf955e2c85b6107bb532ad1bc2900bcbefd3017f31dc12905665b28800b615a1" exitCode=60 Oct 30 06:15:20.211040 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.211020 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" event={"ID":"67ba38f1-1102-4702-b073-f80c906b1363","Type":"ContainerDied","Data":"bf955e2c85b6107bb532ad1bc2900bcbefd3017f31dc12905665b28800b615a1"} Oct 30 06:15:20.211097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.211082 5001 scope.go:122] "RemoveContainer" containerID="32685e481278b4704155361d734e72a60b13d26ee5258732babc72e2ec48759f" Oct 30 06:15:20.211585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.211564 5001 scope.go:122] "RemoveContainer" containerID="bf955e2c85b6107bb532ad1bc2900bcbefd3017f31dc12905665b28800b615a1" Oct 30 06:15:20.212389 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:20.212357 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql_odh-ai-gateway-infra(67ba38f1-1102-4702-b073-f80c906b1363)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" podUID="67ba38f1-1102-4702-b073-f80c906b1363" Oct 30 06:15:20.220655 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.220627 5001 generic.go:356] "Generic (PLEG): container finished" podID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerID="ff6c95e0d759a1052d5f032debc0eba2167c681c29853717a5ee609da3b4a240" exitCode=60 Oct 30 06:15:20.220749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.220710 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" event={"ID":"6af42bac-496e-4ffa-92a3-64b7c4852a73","Type":"ContainerDied","Data":"ff6c95e0d759a1052d5f032debc0eba2167c681c29853717a5ee609da3b4a240"} Oct 30 06:15:20.221141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.221124 5001 scope.go:122] "RemoveContainer" containerID="ff6c95e0d759a1052d5f032debc0eba2167c681c29853717a5ee609da3b4a240" Oct 30 06:15:20.221517 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:20.221492 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f_odh-ai-gateway-infra(6af42bac-496e-4ffa-92a3-64b7c4852a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" Oct 30 06:15:20.224299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.224277 5001 generic.go:356] "Generic (PLEG): container finished" podID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerID="03b2bb8c2063e1f7bffb7f0210357bcf50bfb2b46f00fd3940fbd1274eab9202" exitCode=60 Oct 30 06:15:20.224924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.224905 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" event={"ID":"79946c52-915f-4a02-bce3-3d4c92c8360b","Type":"ContainerDied","Data":"03b2bb8c2063e1f7bffb7f0210357bcf50bfb2b46f00fd3940fbd1274eab9202"} Oct 30 06:15:20.231703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.231684 5001 scope.go:122] "RemoveContainer" containerID="81fee31854172eea5086ca4aa547dc4213503e51ee5f6d2c04ad13d992289a08" Oct 30 06:15:20.268894 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.268098 5001 scope.go:122] "RemoveContainer" containerID="001c4459a205d4f38613f6e3e6d447f9c6e8934fe1c62568994d1e740e2d5cbf" Oct 30 06:15:20.291058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.291024 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q"] Oct 30 06:15:20.394332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.391643 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr"] Oct 30 06:15:20.414128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.413736 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f"] Oct 30 06:15:20.496097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.496060 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9"] Oct 30 06:15:20.511826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.510265 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql"] Oct 30 06:15:20.548916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.541349 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" Oct 30 06:15:20.612069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.612044 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79946c52-915f-4a02-bce3-3d4c92c8360b-kube-api-access-qzrzp\" (UniqueName: \"kubernetes.io/projected/79946c52-915f-4a02-bce3-3d4c92c8360b-kube-api-access-qzrzp\") pod \"79946c52-915f-4a02-bce3-3d4c92c8360b\" (UID: \"79946c52-915f-4a02-bce3-3d4c92c8360b\") " Oct 30 06:15:20.612421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.612396 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79946c52-915f-4a02-bce3-3d4c92c8360b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79946c52-915f-4a02-bce3-3d4c92c8360b-maas-api-service-ca\") pod \"79946c52-915f-4a02-bce3-3d4c92c8360b\" (UID: \"79946c52-915f-4a02-bce3-3d4c92c8360b\") " Oct 30 06:15:20.613565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.613537 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79946c52-915f-4a02-bce3-3d4c92c8360b-maas-api-service-ca" pod "79946c52-915f-4a02-bce3-3d4c92c8360b" (UID: "79946c52-915f-4a02-bce3-3d4c92c8360b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:20.613694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.613675 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79946c52-915f-4a02-bce3-3d4c92c8360b-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.616534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.616504 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79946c52-915f-4a02-bce3-3d4c92c8360b-kube-api-access-qzrzp" pod "79946c52-915f-4a02-bce3-3d4c92c8360b" (UID: "79946c52-915f-4a02-bce3-3d4c92c8360b"). InnerVolumeSpecName "kube-api-access-qzrzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:20.643535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.643025 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" Oct 30 06:15:20.717518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.715158 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b87dd6d-e591-475a-80b4-389052f0c713-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b87dd6d-e591-475a-80b4-389052f0c713-maas-api-service-ca\") pod \"5b87dd6d-e591-475a-80b4-389052f0c713\" (UID: \"5b87dd6d-e591-475a-80b4-389052f0c713\") " Oct 30 06:15:20.717518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.715424 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b87dd6d-e591-475a-80b4-389052f0c713-kube-api-access-m7wjj\" (UniqueName: \"kubernetes.io/projected/5b87dd6d-e591-475a-80b4-389052f0c713-kube-api-access-m7wjj\") pod \"5b87dd6d-e591-475a-80b4-389052f0c713\" (UID: \"5b87dd6d-e591-475a-80b4-389052f0c713\") " Oct 30 06:15:20.717518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.716147 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzrzp\" (UniqueName: \"kubernetes.io/projected/79946c52-915f-4a02-bce3-3d4c92c8360b-kube-api-access-qzrzp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.719897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.719865 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b87dd6d-e591-475a-80b4-389052f0c713-maas-api-service-ca" pod "5b87dd6d-e591-475a-80b4-389052f0c713" (UID: "5b87dd6d-e591-475a-80b4-389052f0c713"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:20.724721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.724663 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" Oct 30 06:15:20.727887 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.727856 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b87dd6d-e591-475a-80b4-389052f0c713-kube-api-access-m7wjj" pod "5b87dd6d-e591-475a-80b4-389052f0c713" (UID: "5b87dd6d-e591-475a-80b4-389052f0c713"). InnerVolumeSpecName "kube-api-access-m7wjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:20.827330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.817210 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" Oct 30 06:15:20.827330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.817299 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b4a966-5447-45de-ab4a-c79b929a3d78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b4a966-5447-45de-ab4a-c79b929a3d78-maas-api-service-ca\") pod \"43b4a966-5447-45de-ab4a-c79b929a3d78\" (UID: \"43b4a966-5447-45de-ab4a-c79b929a3d78\") " Oct 30 06:15:20.827330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.817479 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b4a966-5447-45de-ab4a-c79b929a3d78-kube-api-access-9w8zf\" (UniqueName: \"kubernetes.io/projected/43b4a966-5447-45de-ab4a-c79b929a3d78-kube-api-access-9w8zf\") pod \"43b4a966-5447-45de-ab4a-c79b929a3d78\" (UID: \"43b4a966-5447-45de-ab4a-c79b929a3d78\") " Oct 30 06:15:20.827330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.818043 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b4a966-5447-45de-ab4a-c79b929a3d78-maas-api-service-ca" pod "43b4a966-5447-45de-ab4a-c79b929a3d78" (UID: "43b4a966-5447-45de-ab4a-c79b929a3d78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:20.827330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.818282 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b87dd6d-e591-475a-80b4-389052f0c713-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.827330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.818298 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7wjj\" (UniqueName: \"kubernetes.io/projected/5b87dd6d-e591-475a-80b4-389052f0c713-kube-api-access-m7wjj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.827330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.818328 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b4a966-5447-45de-ab4a-c79b929a3d78-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.827330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.820460 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b4a966-5447-45de-ab4a-c79b929a3d78-kube-api-access-9w8zf" pod "43b4a966-5447-45de-ab4a-c79b929a3d78" (UID: "43b4a966-5447-45de-ab4a-c79b929a3d78"). InnerVolumeSpecName "kube-api-access-9w8zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:20.919498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.919463 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-kube-api-access-dmgvw\" (UniqueName: \"kubernetes.io/projected/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-kube-api-access-dmgvw\") pod \"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b\" (UID: \"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b\") " Oct 30 06:15:20.919696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.919584 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-maas-api-service-ca\") pod \"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b\" (UID: \"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b\") " Oct 30 06:15:20.919953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.919930 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-maas-api-service-ca" pod "aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" (UID: "aa5aba07-ee9a-4e79-bb19-a83ad37cec3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:20.920346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.920331 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.920415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.920349 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w8zf\" (UniqueName: \"kubernetes.io/projected/43b4a966-5447-45de-ab4a-c79b929a3d78-kube-api-access-9w8zf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.922963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:20.922200 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-kube-api-access-dmgvw" pod "aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" (UID: "aa5aba07-ee9a-4e79-bb19-a83ad37cec3b"). InnerVolumeSpecName "kube-api-access-dmgvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.021744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.021658 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmgvw\" (UniqueName: \"kubernetes.io/projected/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b-kube-api-access-dmgvw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.235891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.235862 5001 generic.go:356] "Generic (PLEG): container finished" podID="592c1065-0e61-4095-8234-5cba81dd1283" containerID="881ccf274fa07218cf3d509c53d25e5e721f90fca54745d74e6ab8c702dedfdf" exitCode=60 Oct 30 06:15:21.236412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.235942 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" event={"ID":"592c1065-0e61-4095-8234-5cba81dd1283","Type":"ContainerDied","Data":"881ccf274fa07218cf3d509c53d25e5e721f90fca54745d74e6ab8c702dedfdf"} Oct 30 06:15:21.236412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.236007 5001 scope.go:122] "RemoveContainer" containerID="3d200f08b540fa64f34a6f2ba33d70e602711c253d0e1c5a6f6e46b230c18ecf" Oct 30 06:15:21.236568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.236480 5001 scope.go:122] "RemoveContainer" containerID="881ccf274fa07218cf3d509c53d25e5e721f90fca54745d74e6ab8c702dedfdf" Oct 30 06:15:21.236808 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:21.236784 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z_odh-ai-gateway-infra(592c1065-0e61-4095-8234-5cba81dd1283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" podUID="592c1065-0e61-4095-8234-5cba81dd1283" Oct 30 06:15:21.239496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.239463 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" Oct 30 06:15:21.239600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.239513 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q" event={"ID":"5b87dd6d-e591-475a-80b4-389052f0c713","Type":"ContainerDied","Data":"358be630feaa48aed8084955a5755b0aef8031a7364d84e8fc3bfd7911b89b75"} Oct 30 06:15:21.241344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.241273 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" Oct 30 06:15:21.241344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.241287 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr" event={"ID":"43b4a966-5447-45de-ab4a-c79b929a3d78","Type":"ContainerDied","Data":"aa5e05271cb0cb4f72a763f9525ad48219eb0c292cdb2adddcf6ee06adf06b1b"} Oct 30 06:15:21.245230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.245211 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" Oct 30 06:15:21.245348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.245253 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9" event={"ID":"aa5aba07-ee9a-4e79-bb19-a83ad37cec3b","Type":"ContainerDied","Data":"e64c840322bccf28790eedad1e3e4c4c56f1fb5fca0c839292a0de32a0cf496c"} Oct 30 06:15:21.247116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.247101 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" Oct 30 06:15:21.247387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.247118 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc" event={"ID":"79946c52-915f-4a02-bce3-3d4c92c8360b","Type":"ContainerDied","Data":"b1c1b4ee5632f9ede45074a1372dba1204f96dd48767a2792d7de4eb3db18773"} Oct 30 06:15:21.285913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.285806 5001 scope.go:122] "RemoveContainer" containerID="2655e181d8097226e09b89064938967a55d6472d3e7173aa464ef8d653e2f7fe" Oct 30 06:15:21.288870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.287834 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z"] Oct 30 06:15:21.307876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.307853 5001 scope.go:122] "RemoveContainer" containerID="acbc097710183e8832cdc7c5131f598e7e8cadc664a798d86f9b51b42b06f13c" Oct 30 06:15:21.322921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.322887 5001 scope.go:122] "RemoveContainer" containerID="dcf05a3add189c886a5ac855e5daf1fc3b39df0531e8b999b6606bbdb3e695c8" Oct 30 06:15:21.340550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.338472 5001 scope.go:122] "RemoveContainer" containerID="03b2bb8c2063e1f7bffb7f0210357bcf50bfb2b46f00fd3940fbd1274eab9202" Oct 30 06:15:21.389992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.389961 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q"] Oct 30 06:15:21.405325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.400125 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889015-lvk9q"] Oct 30 06:15:21.419329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.416337 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc"] Oct 30 06:15:21.447077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.447045 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e6df62-29889015-ptnfc"] Oct 30 06:15:21.490029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.489993 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr"] Oct 30 06:15:21.495732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.495705 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-x9tnr"] Oct 30 06:15:21.712329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.711557 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9"] Oct 30 06:15:21.712329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.711627 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c15d3a-29889015-598h9"] Oct 30 06:15:21.714497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.713552 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" Oct 30 06:15:21.770004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.769965 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ba38f1-1102-4702-b073-f80c906b1363-kube-api-access-lfgvv\" (UniqueName: \"kubernetes.io/projected/67ba38f1-1102-4702-b073-f80c906b1363-kube-api-access-lfgvv\") pod \"67ba38f1-1102-4702-b073-f80c906b1363\" (UID: \"67ba38f1-1102-4702-b073-f80c906b1363\") " Oct 30 06:15:21.770370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.770345 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ba38f1-1102-4702-b073-f80c906b1363-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ba38f1-1102-4702-b073-f80c906b1363-maas-api-service-ca\") pod \"67ba38f1-1102-4702-b073-f80c906b1363\" (UID: \"67ba38f1-1102-4702-b073-f80c906b1363\") " Oct 30 06:15:21.789651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.789568 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ba38f1-1102-4702-b073-f80c906b1363-maas-api-service-ca" pod "67ba38f1-1102-4702-b073-f80c906b1363" (UID: "67ba38f1-1102-4702-b073-f80c906b1363"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.793458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.793428 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" Oct 30 06:15:21.794613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.794576 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ba38f1-1102-4702-b073-f80c906b1363-kube-api-access-lfgvv" pod "67ba38f1-1102-4702-b073-f80c906b1363" (UID: "67ba38f1-1102-4702-b073-f80c906b1363"). InnerVolumeSpecName "kube-api-access-lfgvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.844195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.844130 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n"] Oct 30 06:15:21.844924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.844836 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" containerID="cri-o://553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b" gracePeriod=30 Oct 30 06:15:21.877588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.877550 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af42bac-496e-4ffa-92a3-64b7c4852a73-kube-api-access-kw674\" (UniqueName: \"kubernetes.io/projected/6af42bac-496e-4ffa-92a3-64b7c4852a73-kube-api-access-kw674\") pod \"6af42bac-496e-4ffa-92a3-64b7c4852a73\" (UID: \"6af42bac-496e-4ffa-92a3-64b7c4852a73\") " Oct 30 06:15:21.889532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.889507 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af42bac-496e-4ffa-92a3-64b7c4852a73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af42bac-496e-4ffa-92a3-64b7c4852a73-maas-api-service-ca\") pod \"6af42bac-496e-4ffa-92a3-64b7c4852a73\" (UID: \"6af42bac-496e-4ffa-92a3-64b7c4852a73\") " Oct 30 06:15:21.890277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.890258 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfgvv\" (UniqueName: \"kubernetes.io/projected/67ba38f1-1102-4702-b073-f80c906b1363-kube-api-access-lfgvv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.890411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.890398 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ba38f1-1102-4702-b073-f80c906b1363-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.890748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.890733 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af42bac-496e-4ffa-92a3-64b7c4852a73-maas-api-service-ca" pod "6af42bac-496e-4ffa-92a3-64b7c4852a73" (UID: "6af42bac-496e-4ffa-92a3-64b7c4852a73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.891756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:21.891730 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af42bac-496e-4ffa-92a3-64b7c4852a73-kube-api-access-kw674" pod "6af42bac-496e-4ffa-92a3-64b7c4852a73" (UID: "6af42bac-496e-4ffa-92a3-64b7c4852a73"). InnerVolumeSpecName "kube-api-access-kw674". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:22.003913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.003866 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af42bac-496e-4ffa-92a3-64b7c4852a73-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:22.004161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.004140 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw674\" (UniqueName: \"kubernetes.io/projected/6af42bac-496e-4ffa-92a3-64b7c4852a73-kube-api-access-kw674\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:22.179539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.179505 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" path="/var/lib/kubelet/pods/43b4a966-5447-45de-ab4a-c79b929a3d78/volumes" Oct 30 06:15:22.180548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.180524 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" path="/var/lib/kubelet/pods/5b87dd6d-e591-475a-80b4-389052f0c713/volumes" Oct 30 06:15:22.193338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.181254 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" path="/var/lib/kubelet/pods/79946c52-915f-4a02-bce3-3d4c92c8360b/volumes" Oct 30 06:15:22.201987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.199671 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" path="/var/lib/kubelet/pods/aa5aba07-ee9a-4e79-bb19-a83ad37cec3b/volumes" Oct 30 06:15:22.320993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.320957 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" event={"ID":"6af42bac-496e-4ffa-92a3-64b7c4852a73","Type":"ContainerDied","Data":"9cd863705c74963703b975d4fafdb329caebf34b35f692611b5239916da19564"} Oct 30 06:15:22.323328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.323292 5001 scope.go:122] "RemoveContainer" containerID="ff6c95e0d759a1052d5f032debc0eba2167c681c29853717a5ee609da3b4a240" Oct 30 06:15:22.323713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.323684 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f" Oct 30 06:15:22.353422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.353394 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" Oct 30 06:15:22.353791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.353749 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql" event={"ID":"67ba38f1-1102-4702-b073-f80c906b1363","Type":"ContainerDied","Data":"d0200a3fe99ee69a7f114634e68364a197981c9f66716f09e0987855941cf312"} Oct 30 06:15:22.385720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.385595 5001 scope.go:122] "RemoveContainer" containerID="bf955e2c85b6107bb532ad1bc2900bcbefd3017f31dc12905665b28800b615a1" Oct 30 06:15:22.411005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.409708 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f"] Oct 30 06:15:22.418388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.418353 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889015-rff5f"] Oct 30 06:15:22.536393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.536370 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql"] Oct 30 06:15:22.540707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.540676 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ba44b248-29889015-k7gql"] Oct 30 06:15:22.589251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.589226 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" Oct 30 06:15:22.648411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.648376 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592c1065-0e61-4095-8234-5cba81dd1283-kube-api-access-fctxx\" (UniqueName: \"kubernetes.io/projected/592c1065-0e61-4095-8234-5cba81dd1283-kube-api-access-fctxx\") pod \"592c1065-0e61-4095-8234-5cba81dd1283\" (UID: \"592c1065-0e61-4095-8234-5cba81dd1283\") " Oct 30 06:15:22.648634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.648489 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592c1065-0e61-4095-8234-5cba81dd1283-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/592c1065-0e61-4095-8234-5cba81dd1283-maas-api-service-ca\") pod \"592c1065-0e61-4095-8234-5cba81dd1283\" (UID: \"592c1065-0e61-4095-8234-5cba81dd1283\") " Oct 30 06:15:22.649494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.649218 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592c1065-0e61-4095-8234-5cba81dd1283-maas-api-service-ca" pod "592c1065-0e61-4095-8234-5cba81dd1283" (UID: "592c1065-0e61-4095-8234-5cba81dd1283"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:22.649494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.649466 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/592c1065-0e61-4095-8234-5cba81dd1283-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:22.677958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.674620 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592c1065-0e61-4095-8234-5cba81dd1283-kube-api-access-fctxx" pod "592c1065-0e61-4095-8234-5cba81dd1283" (UID: "592c1065-0e61-4095-8234-5cba81dd1283"). InnerVolumeSpecName "kube-api-access-fctxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:22.751229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:22.751152 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fctxx\" (UniqueName: \"kubernetes.io/projected/592c1065-0e61-4095-8234-5cba81dd1283-kube-api-access-fctxx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:23.372031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:23.371999 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" Oct 30 06:15:23.373351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:23.372733 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z" event={"ID":"592c1065-0e61-4095-8234-5cba81dd1283","Type":"ContainerDied","Data":"505660ae676b078c06625838fba2726c764da0f14b49804a7e7cf6e62fdea89f"} Oct 30 06:15:23.373351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:23.372802 5001 scope.go:122] "RemoveContainer" containerID="881ccf274fa07218cf3d509c53d25e5e721f90fca54745d74e6ab8c702dedfdf" Oct 30 06:15:23.441536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:23.441504 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z"] Oct 30 06:15:23.447674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:23.447644 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889015-tcl2z"] Oct 30 06:15:23.912458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:23.907416 5001 patch_prober.go:29] interesting pod/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:15:23.912458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:23.907492 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:15:24.120903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:24.120871 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592c1065-0e61-4095-8234-5cba81dd1283" path="/var/lib/kubelet/pods/592c1065-0e61-4095-8234-5cba81dd1283/volumes" Oct 30 06:15:24.121512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:24.121492 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ba38f1-1102-4702-b073-f80c906b1363" path="/var/lib/kubelet/pods/67ba38f1-1102-4702-b073-f80c906b1363/volumes" Oct 30 06:15:24.122041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:24.122025 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" path="/var/lib/kubelet/pods/6af42bac-496e-4ffa-92a3-64b7c4852a73/volumes" Oct 30 06:15:27.069714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.069684 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:15:27.157737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.157638 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-envoy\") pod \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " Oct 30 06:15:27.157914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.157801 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-kube-api-access-t9bnc\" (UniqueName: \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-kube-api-access-t9bnc\") pod \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " Oct 30 06:15:27.157914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.157838 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-credential-socket\") pod \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " Oct 30 06:15:27.158041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.157996 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-certs\") pod \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " Oct 30 06:15:27.158041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.158031 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istiod-ca-cert\") pod \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " Oct 30 06:15:27.158153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.158098 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-podinfo\") pod \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " Oct 30 06:15:27.158210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.158157 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-socket\") pod \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " Oct 30 06:15:27.158210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.158199 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-data\") pod \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " Oct 30 06:15:27.158370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.158295 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-token\" (UniqueName: \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-token\") pod \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\" (UID: \"f43aeac6-adbb-433c-92b2-3d92cb5c9660\") " Oct 30 06:15:27.160010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.158796 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istiod-ca-cert" pod "f43aeac6-adbb-433c-92b2-3d92cb5c9660" (UID: "f43aeac6-adbb-433c-92b2-3d92cb5c9660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:27.160010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.159087 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:27.160010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.159394 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-credential-socket" pod "f43aeac6-adbb-433c-92b2-3d92cb5c9660" (UID: "f43aeac6-adbb-433c-92b2-3d92cb5c9660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:27.160010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.159578 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-certs" pod "f43aeac6-adbb-433c-92b2-3d92cb5c9660" (UID: "f43aeac6-adbb-433c-92b2-3d92cb5c9660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:27.160010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.159744 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-socket" pod "f43aeac6-adbb-433c-92b2-3d92cb5c9660" (UID: "f43aeac6-adbb-433c-92b2-3d92cb5c9660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:27.166252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.161348 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-data" pod "f43aeac6-adbb-433c-92b2-3d92cb5c9660" (UID: "f43aeac6-adbb-433c-92b2-3d92cb5c9660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:27.166252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.161536 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-token" pod "f43aeac6-adbb-433c-92b2-3d92cb5c9660" (UID: "f43aeac6-adbb-433c-92b2-3d92cb5c9660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:27.166552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.166511 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-podinfo" pod "f43aeac6-adbb-433c-92b2-3d92cb5c9660" (UID: "f43aeac6-adbb-433c-92b2-3d92cb5c9660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:15:27.167709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.167662 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-envoy" pod "f43aeac6-adbb-433c-92b2-3d92cb5c9660" (UID: "f43aeac6-adbb-433c-92b2-3d92cb5c9660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:27.169914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.169885 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-kube-api-access-t9bnc" pod "f43aeac6-adbb-433c-92b2-3d92cb5c9660" (UID: "f43aeac6-adbb-433c-92b2-3d92cb5c9660"). InnerVolumeSpecName "kube-api-access-t9bnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:27.260664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.260637 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9bnc\" (UniqueName: \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-kube-api-access-t9bnc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:27.260664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.260662 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:27.260877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.260680 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:27.260877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.260689 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:27.260877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.260697 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:27.260877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.260705 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:27.260877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.260713 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:27.260877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.260721 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43aeac6-adbb-433c-92b2-3d92cb5c9660-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:27.449405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.447612 5001 generic.go:356] "Generic (PLEG): container finished" podID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerID="553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b" exitCode=0 Oct 30 06:15:27.449405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.447740 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" event={"ID":"f43aeac6-adbb-433c-92b2-3d92cb5c9660","Type":"ContainerDied","Data":"553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b"} Oct 30 06:15:27.449405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.447791 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" event={"ID":"f43aeac6-adbb-433c-92b2-3d92cb5c9660","Type":"ContainerDied","Data":"7aba70999081b924e3382f1059d6116b3ed39dbcd5699b27dd1df9c841819fc4"} Oct 30 06:15:27.449405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.447819 5001 scope.go:122] "RemoveContainer" containerID="553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b" Oct 30 06:15:27.449405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.448066 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n" Oct 30 06:15:27.489565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.489530 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" event={"ID":"24b399b8-2225-4498-aa8f-e5ea451f7131","Type":"ContainerDied","Data":"e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0"} Oct 30 06:15:27.489763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.489551 5001 generic.go:356] "Generic (PLEG): container finished" podID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerID="e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0" exitCode=0 Oct 30 06:15:27.490188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.489966 5001 scope.go:122] "RemoveContainer" containerID="e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0" Oct 30 06:15:27.494693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.494463 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:27.494693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.494560 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:27.494693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.494614 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:27.503870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.503849 5001 scope.go:122] "RemoveContainer" containerID="e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a" Oct 30 06:15:27.519631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.519605 5001 scope.go:122] "RemoveContainer" containerID="553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b" Oct 30 06:15:27.519979 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:27.519958 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b\": container with ID starting with 553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b not found: ID does not exist" containerID="553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b" Oct 30 06:15:27.520032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.519991 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b"} err="failed to get container status \"553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b\": rpc error: code = NotFound desc = could not find container \"553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b\": container with ID starting with 553ffc774d11185925ca033ea8e394ebc3ef6fdb39c5e3a55204727054b4e05b not found: ID does not exist" Oct 30 06:15:27.520032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.520009 5001 scope.go:122] "RemoveContainer" containerID="e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a" Oct 30 06:15:27.520249 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:27.520233 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a\": container with ID starting with e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a not found: ID does not exist" containerID="e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a" Oct 30 06:15:27.520299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.520254 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a"} err="failed to get container status \"e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a\": rpc error: code = NotFound desc = could not find container \"e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a\": container with ID starting with e4a12d83b7825bb0f04cb2f91881931d3836f969b0131180fcfccf63a2d0fa9a not found: ID does not exist" Oct 30 06:15:27.529994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.529971 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n"] Oct 30 06:15:27.534786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:27.534759 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f789d7-openshift-default-6f89948549-8zh7n"] Oct 30 06:15:28.127096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:28.127065 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" path="/var/lib/kubelet/pods/f43aeac6-adbb-433c-92b2-3d92cb5c9660/volumes" Oct 30 06:15:28.504478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:28.504444 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" event={"ID":"24b399b8-2225-4498-aa8f-e5ea451f7131","Type":"ContainerStarted","Data":"b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d"} Oct 30 06:15:29.133905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:29.133876 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:15:29.134354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:29.134037 5001 patch_prober.go:29] interesting pod/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:15:29.134354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:29.134077 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 06:15:30.120357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.120204 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn"] Oct 30 06:15:30.120934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.120914 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.120936 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.120949 5001 cpu_manager.go:414] "containerMap: removing container" podUID="67ba38f1-1102-4702-b073-f80c906b1363" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.120956 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ba38f1-1102-4702-b073-f80c906b1363" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.120970 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.120977 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.120999 5001 cpu_manager.go:414] "containerMap: removing container" podUID="592c1065-0e61-4095-8234-5cba81dd1283" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121005 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592c1065-0e61-4095-8234-5cba81dd1283" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121018 5001 cpu_manager.go:414] "containerMap: removing container" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121024 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121036 5001 cpu_manager.go:414] "containerMap: removing container" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121042 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121062 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121067 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" containerName="cleanup" Oct 30 06:15:30.121074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121079 5001 cpu_manager.go:414] "containerMap: removing container" podUID="67ba38f1-1102-4702-b073-f80c906b1363" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121085 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ba38f1-1102-4702-b073-f80c906b1363" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121095 5001 cpu_manager.go:414] "containerMap: removing container" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121102 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121125 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121132 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121143 5001 cpu_manager.go:414] "containerMap: removing container" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121150 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121164 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121170 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121185 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121191 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121209 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121216 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121224 5001 cpu_manager.go:414] "containerMap: removing container" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121231 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121246 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121254 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121274 5001 cpu_manager.go:414] "containerMap: removing container" podUID="592c1065-0e61-4095-8234-5cba81dd1283" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121281 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592c1065-0e61-4095-8234-5cba81dd1283" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121293 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121300 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121335 5001 cpu_manager.go:414] "containerMap: removing container" podUID="592c1065-0e61-4095-8234-5cba81dd1283" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121342 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592c1065-0e61-4095-8234-5cba81dd1283" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121508 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="592c1065-0e61-4095-8234-5cba81dd1283" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121523 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121533 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="592c1065-0e61-4095-8234-5cba81dd1283" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121542 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121549 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ba38f1-1102-4702-b073-f80c906b1363" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121559 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="592c1065-0e61-4095-8234-5cba81dd1283" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121569 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121576 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121587 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121595 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ba38f1-1102-4702-b073-f80c906b1363" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121606 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121617 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121626 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121635 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121644 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121653 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121662 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121673 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b4a966-5447-45de-ab4a-c79b929a3d78" containerName="cleanup" Oct 30 06:15:30.121878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.121683 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" containerName="cleanup" Oct 30 06:15:30.126245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.126224 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn"] Oct 30 06:15:30.126397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.126338 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:30.134420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.133813 5001 patch_prober.go:29] interesting pod/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:15:30.134420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.133882 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 06:15:30.141978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.141951 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:15:30.183382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.176543 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:15:30.183382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.177451 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:15:30.183382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.178089 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:15:30.229277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.229253 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:15:30.233598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.233578 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:30.233679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.233647 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:30.233753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.233733 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzhg5\" (UniqueName: \"kubernetes.io/projected/acd5a5be-37ed-437a-b707-d3ae59c793af-kube-api-access-kzhg5\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:30.335176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.335142 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:30.335401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.335222 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:30.335401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.335257 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzhg5\" (UniqueName: \"kubernetes.io/projected/acd5a5be-37ed-437a-b707-d3ae59c793af-kube-api-access-kzhg5\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:30.336001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.335980 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:30.337728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.337706 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:30.392387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.392288 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzhg5\" (UniqueName: \"kubernetes.io/projected/acd5a5be-37ed-437a-b707-d3ae59c793af-kube-api-access-kzhg5\") pod \"maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:30.441564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.441532 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:30.796244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:30.796171 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn"] Oct 30 06:15:31.134341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:31.134285 5001 patch_prober.go:29] interesting pod/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:15:31.134499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:31.134362 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 06:15:31.534895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:31.534797 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" event={"ID":"acd5a5be-37ed-437a-b707-d3ae59c793af","Type":"ContainerStarted","Data":"e09f04da5a808e1694a2a0b01c9b13d6005e72009e969e1ed347bf71675072eb"} Oct 30 06:15:31.534895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:31.534862 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" event={"ID":"acd5a5be-37ed-437a-b707-d3ae59c793af","Type":"ContainerStarted","Data":"8217d481cccd50d96feb20e682b918f2701f560f8222c98aa41189d676ddd15d"} Oct 30 06:15:31.585433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:31.579858 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" podStartSLOduration=1.579839956 podStartE2EDuration="1.579839956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:31.578538063 +0000 UTC m=+1345.574495074" watchObservedRunningTime="2026-10-30 06:15:31.579839956 +0000 UTC m=+1345.575796968" Oct 30 06:15:32.133699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:32.133622 5001 patch_prober.go:29] interesting pod/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:15:32.133699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:32.133671 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 06:15:32.545005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:32.544917 5001 generic.go:356] "Generic (PLEG): container finished" podID="acd5a5be-37ed-437a-b707-d3ae59c793af" containerID="e09f04da5a808e1694a2a0b01c9b13d6005e72009e969e1ed347bf71675072eb" exitCode=0 Oct 30 06:15:32.545468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:32.544999 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" event={"ID":"acd5a5be-37ed-437a-b707-d3ae59c793af","Type":"ContainerDied","Data":"e09f04da5a808e1694a2a0b01c9b13d6005e72009e969e1ed347bf71675072eb"} Oct 30 06:15:33.133507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:33.133470 5001 patch_prober.go:29] interesting pod/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:15:33.133690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:33.133535 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 06:15:33.836701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:33.836633 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:33.936839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:33.933396 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-xks-ca\") pod \"acd5a5be-37ed-437a-b707-d3ae59c793af\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " Oct 30 06:15:33.936839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:33.933618 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-openshift-ca\") pod \"acd5a5be-37ed-437a-b707-d3ae59c793af\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " Oct 30 06:15:33.936839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:33.933883 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd5a5be-37ed-437a-b707-d3ae59c793af-kube-api-access-kzhg5\" (UniqueName: \"kubernetes.io/projected/acd5a5be-37ed-437a-b707-d3ae59c793af-kube-api-access-kzhg5\") pod \"acd5a5be-37ed-437a-b707-d3ae59c793af\" (UID: \"acd5a5be-37ed-437a-b707-d3ae59c793af\") " Oct 30 06:15:33.951582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:33.951518 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd5a5be-37ed-437a-b707-d3ae59c793af-kube-api-access-kzhg5" pod "acd5a5be-37ed-437a-b707-d3ae59c793af" (UID: "acd5a5be-37ed-437a-b707-d3ae59c793af"). InnerVolumeSpecName "kube-api-access-kzhg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:33.952150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:33.952124 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-openshift-ca" pod "acd5a5be-37ed-437a-b707-d3ae59c793af" (UID: "acd5a5be-37ed-437a-b707-d3ae59c793af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:33.967900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:33.967863 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-xks-ca" pod "acd5a5be-37ed-437a-b707-d3ae59c793af" (UID: "acd5a5be-37ed-437a-b707-d3ae59c793af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:34.036062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:34.035999 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:34.036062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:34.036030 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzhg5\" (UniqueName: \"kubernetes.io/projected/acd5a5be-37ed-437a-b707-d3ae59c793af-kube-api-access-kzhg5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:34.036062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:34.036040 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd5a5be-37ed-437a-b707-d3ae59c793af-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:34.133563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:34.133512 5001 patch_prober.go:29] interesting pod/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:15:34.133692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:34.133663 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 06:15:34.597285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:34.597195 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" event={"ID":"acd5a5be-37ed-437a-b707-d3ae59c793af","Type":"ContainerDied","Data":"8217d481cccd50d96feb20e682b918f2701f560f8222c98aa41189d676ddd15d"} Oct 30 06:15:34.597575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:34.597543 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8217d481cccd50d96feb20e682b918f2701f560f8222c98aa41189d676ddd15d" Oct 30 06:15:34.597714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:34.597642 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn" Oct 30 06:15:35.149028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:35.148994 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:15:35.151186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:35.151157 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:15:35.152364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:35.152291 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:15:38.300869 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:38.298411 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:15:39.329206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:39.329162 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb"] Oct 30 06:15:39.336890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:39.336825 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6df62-96mrb"] Oct 30 06:15:40.045950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:40.045916 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc"] Oct 30 06:15:40.060956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:40.060922 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de5a995-rnbmc"] Oct 30 06:15:40.121531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:40.121493 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ccc380-344b-42c3-a5ca-d31b5c26cbd8" path="/var/lib/kubelet/pods/97ccc380-344b-42c3-a5ca-d31b5c26cbd8/volumes" Oct 30 06:15:40.122132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:40.122112 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3164cd0-0181-4645-9332-f82110a5e5c9" path="/var/lib/kubelet/pods/d3164cd0-0181-4645-9332-f82110a5e5c9/volumes" Oct 30 06:15:40.553758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:40.553720 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc"] Oct 30 06:15:40.611366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:40.611288 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26"] Oct 30 06:15:45.591331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:45.568097 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" podUID="7a1a460b-d6a0-48e4-9fd2-b82306b37b2e" containerName="payload-processing" containerID="cri-o://d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982" gracePeriod=25 Oct 30 06:15:45.632442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:45.619408 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" podUID="640bf110-4c9a-477f-bb8e-fb32739309ce" containerName="payload-pre-processing" containerID="cri-o://8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37" gracePeriod=25 Oct 30 06:15:45.973343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:45.969558 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:15:46.034907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.034875 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:15:46.073473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.073445 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-kube-api-access-7plph\" (UniqueName: \"kubernetes.io/projected/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-kube-api-access-7plph\") pod \"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e\" (UID: \"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e\") " Oct 30 06:15:46.073676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.073524 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-plugins-config-volume\") pod \"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e\" (UID: \"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e\") " Oct 30 06:15:46.074219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.074147 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-plugins-config-volume" pod "7a1a460b-d6a0-48e4-9fd2-b82306b37b2e" (UID: "7a1a460b-d6a0-48e4-9fd2-b82306b37b2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:46.074423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.074401 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:46.078460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.078428 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-kube-api-access-7plph" pod "7a1a460b-d6a0-48e4-9fd2-b82306b37b2e" (UID: "7a1a460b-d6a0-48e4-9fd2-b82306b37b2e"). InnerVolumeSpecName "kube-api-access-7plph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:46.160369 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:46.160330 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1a460b_d6a0_48e4_9fd2_b82306b37b2e.slice\": RecentStats: unable to find data in memory cache]" Oct 30 06:15:46.175793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.175762 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/640bf110-4c9a-477f-bb8e-fb32739309ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640bf110-4c9a-477f-bb8e-fb32739309ce-plugins-config-volume\") pod \"640bf110-4c9a-477f-bb8e-fb32739309ce\" (UID: \"640bf110-4c9a-477f-bb8e-fb32739309ce\") " Oct 30 06:15:46.176034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.176018 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640bf110-4c9a-477f-bb8e-fb32739309ce-kube-api-access-cmpln\" (UniqueName: \"kubernetes.io/projected/640bf110-4c9a-477f-bb8e-fb32739309ce-kube-api-access-cmpln\") pod \"640bf110-4c9a-477f-bb8e-fb32739309ce\" (UID: \"640bf110-4c9a-477f-bb8e-fb32739309ce\") " Oct 30 06:15:46.176131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.176107 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/640bf110-4c9a-477f-bb8e-fb32739309ce-plugins-config-volume" pod "640bf110-4c9a-477f-bb8e-fb32739309ce" (UID: "640bf110-4c9a-477f-bb8e-fb32739309ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:46.176665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.176644 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640bf110-4c9a-477f-bb8e-fb32739309ce-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:46.176767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.176671 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7plph\" (UniqueName: \"kubernetes.io/projected/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e-kube-api-access-7plph\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:46.178367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.178329 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640bf110-4c9a-477f-bb8e-fb32739309ce-kube-api-access-cmpln" pod "640bf110-4c9a-477f-bb8e-fb32739309ce" (UID: "640bf110-4c9a-477f-bb8e-fb32739309ce"). InnerVolumeSpecName "kube-api-access-cmpln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:46.278068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.278032 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmpln\" (UniqueName: \"kubernetes.io/projected/640bf110-4c9a-477f-bb8e-fb32739309ce-kube-api-access-cmpln\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:46.738041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.738003 5001 generic.go:356] "Generic (PLEG): container finished" podID="7a1a460b-d6a0-48e4-9fd2-b82306b37b2e" containerID="d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982" exitCode=0 Oct 30 06:15:46.738551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.738053 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" Oct 30 06:15:46.738551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.738086 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" event={"ID":"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e","Type":"ContainerDied","Data":"d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982"} Oct 30 06:15:46.738551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.738153 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc" event={"ID":"7a1a460b-d6a0-48e4-9fd2-b82306b37b2e","Type":"ContainerDied","Data":"099a9a4583ceaec6c4107f96b801274f50dfc8ed42e9fef9c496537e244d5430"} Oct 30 06:15:46.738551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.738185 5001 scope.go:122] "RemoveContainer" containerID="d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982" Oct 30 06:15:46.739961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.739938 5001 generic.go:356] "Generic (PLEG): container finished" podID="640bf110-4c9a-477f-bb8e-fb32739309ce" containerID="8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37" exitCode=0 Oct 30 06:15:46.740072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.739988 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" Oct 30 06:15:46.740136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.740082 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" event={"ID":"640bf110-4c9a-477f-bb8e-fb32739309ce","Type":"ContainerDied","Data":"8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37"} Oct 30 06:15:46.740188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.740137 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26" event={"ID":"640bf110-4c9a-477f-bb8e-fb32739309ce","Type":"ContainerDied","Data":"64d235602e8275e59c47d3057595f80d07a582c3e87fc9721ae6c5763ff3ec5e"} Oct 30 06:15:46.751418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.751394 5001 scope.go:122] "RemoveContainer" containerID="d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982" Oct 30 06:15:46.751793 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:46.751773 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982\": container with ID starting with d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982 not found: ID does not exist" containerID="d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982" Oct 30 06:15:46.751879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.751805 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982"} err="failed to get container status \"d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982\": rpc error: code = NotFound desc = could not find container \"d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982\": container with ID starting with d3c84c5f0066ca4e9c18bf76de65903ae20c731cda856ccd773525f4d0aa2982 not found: ID does not exist" Oct 30 06:15:46.751879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.751828 5001 scope.go:122] "RemoveContainer" containerID="8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37" Oct 30 06:15:46.765453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.765431 5001 scope.go:122] "RemoveContainer" containerID="8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37" Oct 30 06:15:46.765757 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:46.765737 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37\": container with ID starting with 8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37 not found: ID does not exist" containerID="8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37" Oct 30 06:15:46.765818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.765767 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37"} err="failed to get container status \"8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37\": rpc error: code = NotFound desc = could not find container \"8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37\": container with ID starting with 8d55da8b17405c838eb7456d20cc7f2dd71677cee20a87e8d39dcd9314e35a37 not found: ID does not exist" Oct 30 06:15:46.795638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.795608 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc"] Oct 30 06:15:46.808293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.808263 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ba44b248-868467698b-2z4nc"] Oct 30 06:15:46.840574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.840537 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26"] Oct 30 06:15:46.849244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.849212 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ba44b248-d59758b87-4cn26"] Oct 30 06:15:46.884358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.884323 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl"] Oct 30 06:15:46.884547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:46.884526 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" containerID="cri-o://cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d" gracePeriod=30 Oct 30 06:15:47.367372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.361275 5001 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:15:47.367372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.361364 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:15:47.504354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.504319 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk"] Oct 30 06:15:47.505007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.504994 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7a1a460b-d6a0-48e4-9fd2-b82306b37b2e" containerName="payload-processing" Oct 30 06:15:47.505054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505009 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1a460b-d6a0-48e4-9fd2-b82306b37b2e" containerName="payload-processing" Oct 30 06:15:47.505054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505026 5001 cpu_manager.go:414] "containerMap: removing container" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerName="cleanup" Oct 30 06:15:47.505054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505032 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79946c52-915f-4a02-bce3-3d4c92c8360b" containerName="cleanup" Oct 30 06:15:47.505149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505061 5001 cpu_manager.go:414] "containerMap: removing container" podUID="acd5a5be-37ed-437a-b707-d3ae59c793af" containerName="revoke-keys" Oct 30 06:15:47.505149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505066 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd5a5be-37ed-437a-b707-d3ae59c793af" containerName="revoke-keys" Oct 30 06:15:47.505149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505082 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerName="cleanup" Oct 30 06:15:47.505149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505087 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5aba07-ee9a-4e79-bb19-a83ad37cec3b" containerName="cleanup" Oct 30 06:15:47.505149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505106 5001 cpu_manager.go:414] "containerMap: removing container" podUID="640bf110-4c9a-477f-bb8e-fb32739309ce" containerName="payload-pre-processing" Oct 30 06:15:47.505149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505111 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640bf110-4c9a-477f-bb8e-fb32739309ce" containerName="payload-pre-processing" Oct 30 06:15:47.505149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505126 5001 cpu_manager.go:414] "containerMap: removing container" podUID="67ba38f1-1102-4702-b073-f80c906b1363" containerName="cleanup" Oct 30 06:15:47.505149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505130 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ba38f1-1102-4702-b073-f80c906b1363" containerName="cleanup" Oct 30 06:15:47.505149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505141 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerName="cleanup" Oct 30 06:15:47.505149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505146 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerName="cleanup" Oct 30 06:15:47.505478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505237 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b87dd6d-e591-475a-80b4-389052f0c713" containerName="cleanup" Oct 30 06:15:47.505478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505254 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="640bf110-4c9a-477f-bb8e-fb32739309ce" containerName="payload-pre-processing" Oct 30 06:15:47.505478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505262 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af42bac-496e-4ffa-92a3-64b7c4852a73" containerName="cleanup" Oct 30 06:15:47.505478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505269 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43aeac6-adbb-433c-92b2-3d92cb5c9660" containerName="istio-proxy" Oct 30 06:15:47.505478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505277 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1a460b-d6a0-48e4-9fd2-b82306b37b2e" containerName="payload-processing" Oct 30 06:15:47.505478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505283 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ba38f1-1102-4702-b073-f80c906b1363" containerName="cleanup" Oct 30 06:15:47.505478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.505290 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd5a5be-37ed-437a-b707-d3ae59c793af" containerName="revoke-keys" Oct 30 06:15:47.510467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.510447 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:47.528492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.526976 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:15:47.528492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.527018 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:15:47.529864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.529841 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk"] Oct 30 06:15:47.599953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.599923 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:15:47.599953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.599940 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:15:47.613886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.613858 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe108e-c25nk\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:47.614042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.613916 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe108e-c25nk\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:47.614084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.614038 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4j6b\" (UniqueName: \"kubernetes.io/projected/f04d0288-0153-4399-bf11-0a137ae83450-kube-api-access-g4j6b\") pod \"maas-api-revoke-keys-e2e-api-b-fe108e-c25nk\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:47.715784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.715705 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe108e-c25nk\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:47.715784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.715763 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe108e-c25nk\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:47.716043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.715802 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4j6b\" (UniqueName: \"kubernetes.io/projected/f04d0288-0153-4399-bf11-0a137ae83450-kube-api-access-g4j6b\") pod \"maas-api-revoke-keys-e2e-api-b-fe108e-c25nk\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:47.716613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.716588 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe108e-c25nk\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:47.718351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.718295 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe108e-c25nk\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:47.741392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.741362 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4j6b\" (UniqueName: \"kubernetes.io/projected/f04d0288-0153-4399-bf11-0a137ae83450-kube-api-access-g4j6b\") pod \"maas-api-revoke-keys-e2e-api-b-fe108e-c25nk\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:47.825903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:47.825871 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:48.120715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:48.120680 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="640bf110-4c9a-477f-bb8e-fb32739309ce" path="/var/lib/kubelet/pods/640bf110-4c9a-477f-bb8e-fb32739309ce/volumes" Oct 30 06:15:48.121129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:48.121114 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a1a460b-d6a0-48e4-9fd2-b82306b37b2e" path="/var/lib/kubelet/pods/7a1a460b-d6a0-48e4-9fd2-b82306b37b2e/volumes" Oct 30 06:15:48.145900 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:15:48.145865 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04d0288_0153_4399_bf11_0a137ae83450.slice/crio-5455dea5515e2543b82c89fc86b4b74c8e3f1d39091f5ff76fe5dff08783cc85 WatchSource:0}: Error finding container 5455dea5515e2543b82c89fc86b4b74c8e3f1d39091f5ff76fe5dff08783cc85: Status 404 returned error can't find the container with id 5455dea5515e2543b82c89fc86b4b74c8e3f1d39091f5ff76fe5dff08783cc85 Oct 30 06:15:48.146041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:48.145944 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk"] Oct 30 06:15:48.764783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:48.764697 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" event={"ID":"f04d0288-0153-4399-bf11-0a137ae83450","Type":"ContainerStarted","Data":"2c050295a0d893e8d10ea77c8c9da72e9b77614accf4ab9dd5d2a7359295acb7"} Oct 30 06:15:48.764783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:48.764752 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" event={"ID":"f04d0288-0153-4399-bf11-0a137ae83450","Type":"ContainerStarted","Data":"5455dea5515e2543b82c89fc86b4b74c8e3f1d39091f5ff76fe5dff08783cc85"} Oct 30 06:15:48.807822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:48.807772 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" podStartSLOduration=1.807758011 podStartE2EDuration="1.807758011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:48.806352243 +0000 UTC m=+1362.802309253" watchObservedRunningTime="2026-10-30 06:15:48.807758011 +0000 UTC m=+1362.803715020" Oct 30 06:15:49.775568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:49.775535 5001 generic.go:356] "Generic (PLEG): container finished" podID="f04d0288-0153-4399-bf11-0a137ae83450" containerID="2c050295a0d893e8d10ea77c8c9da72e9b77614accf4ab9dd5d2a7359295acb7" exitCode=0 Oct 30 06:15:49.775953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:49.775620 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" event={"ID":"f04d0288-0153-4399-bf11-0a137ae83450","Type":"ContainerDied","Data":"2c050295a0d893e8d10ea77c8c9da72e9b77614accf4ab9dd5d2a7359295acb7"} Oct 30 06:15:51.034352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.020895 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:51.102714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.102624 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-xks-ca\") pod \"f04d0288-0153-4399-bf11-0a137ae83450\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " Oct 30 06:15:51.102882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.102750 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04d0288-0153-4399-bf11-0a137ae83450-kube-api-access-g4j6b\" (UniqueName: \"kubernetes.io/projected/f04d0288-0153-4399-bf11-0a137ae83450-kube-api-access-g4j6b\") pod \"f04d0288-0153-4399-bf11-0a137ae83450\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " Oct 30 06:15:51.102882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.102821 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-openshift-ca\") pod \"f04d0288-0153-4399-bf11-0a137ae83450\" (UID: \"f04d0288-0153-4399-bf11-0a137ae83450\") " Oct 30 06:15:51.103957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.103931 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-openshift-ca" pod "f04d0288-0153-4399-bf11-0a137ae83450" (UID: "f04d0288-0153-4399-bf11-0a137ae83450"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:51.105702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.105666 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-xks-ca" pod "f04d0288-0153-4399-bf11-0a137ae83450" (UID: "f04d0288-0153-4399-bf11-0a137ae83450"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:51.106826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.106803 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04d0288-0153-4399-bf11-0a137ae83450-kube-api-access-g4j6b" pod "f04d0288-0153-4399-bf11-0a137ae83450" (UID: "f04d0288-0153-4399-bf11-0a137ae83450"). InnerVolumeSpecName "kube-api-access-g4j6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:51.205708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.205663 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4j6b\" (UniqueName: \"kubernetes.io/projected/f04d0288-0153-4399-bf11-0a137ae83450-kube-api-access-g4j6b\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:51.205708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.205697 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:51.205708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.205712 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04d0288-0153-4399-bf11-0a137ae83450-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:51.799959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.799918 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" Oct 30 06:15:51.799959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.799940 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk" event={"ID":"f04d0288-0153-4399-bf11-0a137ae83450","Type":"ContainerDied","Data":"5455dea5515e2543b82c89fc86b4b74c8e3f1d39091f5ff76fe5dff08783cc85"} Oct 30 06:15:51.800419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:51.799994 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5455dea5515e2543b82c89fc86b4b74c8e3f1d39091f5ff76fe5dff08783cc85" Oct 30 06:15:52.040400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.040361 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4"] Oct 30 06:15:52.044867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.044833 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-982aaa4c-jhzm4"] Oct 30 06:15:52.125759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.125721 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388b8fa3-e252-411f-8e88-6a12feb86d69" path="/var/lib/kubelet/pods/388b8fa3-e252-411f-8e88-6a12feb86d69/volumes" Oct 30 06:15:52.189504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.189480 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:15:52.324600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.324516 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-socket\") pod \"f67c14bb-f110-45db-b9d8-4800a84b7277\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " Oct 30 06:15:52.324781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.324618 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-token\" (UniqueName: \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-token\") pod \"f67c14bb-f110-45db-b9d8-4800a84b7277\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " Oct 30 06:15:52.324781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.324656 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-data\") pod \"f67c14bb-f110-45db-b9d8-4800a84b7277\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " Oct 30 06:15:52.324781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.324688 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67c14bb-f110-45db-b9d8-4800a84b7277-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67c14bb-f110-45db-b9d8-4800a84b7277-istiod-ca-cert\") pod \"f67c14bb-f110-45db-b9d8-4800a84b7277\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " Oct 30 06:15:52.324781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.324718 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-kube-api-access-9g8x7\" (UniqueName: \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-kube-api-access-9g8x7\") pod \"f67c14bb-f110-45db-b9d8-4800a84b7277\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " Oct 30 06:15:52.325006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.324798 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-socket" pod "f67c14bb-f110-45db-b9d8-4800a84b7277" (UID: "f67c14bb-f110-45db-b9d8-4800a84b7277"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.325006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.324859 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-envoy\") pod \"f67c14bb-f110-45db-b9d8-4800a84b7277\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " Oct 30 06:15:52.325006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.324949 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-certs\") pod \"f67c14bb-f110-45db-b9d8-4800a84b7277\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " Oct 30 06:15:52.325006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.324972 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-data" pod "f67c14bb-f110-45db-b9d8-4800a84b7277" (UID: "f67c14bb-f110-45db-b9d8-4800a84b7277"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.325218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.325026 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-credential-socket\") pod \"f67c14bb-f110-45db-b9d8-4800a84b7277\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " Oct 30 06:15:52.325218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.325055 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-podinfo\") pod \"f67c14bb-f110-45db-b9d8-4800a84b7277\" (UID: \"f67c14bb-f110-45db-b9d8-4800a84b7277\") " Oct 30 06:15:52.325218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.325098 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67c14bb-f110-45db-b9d8-4800a84b7277-istiod-ca-cert" pod "f67c14bb-f110-45db-b9d8-4800a84b7277" (UID: "f67c14bb-f110-45db-b9d8-4800a84b7277"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:52.325218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.325208 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-certs" pod "f67c14bb-f110-45db-b9d8-4800a84b7277" (UID: "f67c14bb-f110-45db-b9d8-4800a84b7277"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.325482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.325438 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-credential-socket" pod "f67c14bb-f110-45db-b9d8-4800a84b7277" (UID: "f67c14bb-f110-45db-b9d8-4800a84b7277"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.325772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.325728 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.325772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.325752 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.325772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.325766 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.325973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.325781 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67c14bb-f110-45db-b9d8-4800a84b7277-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.325973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.325793 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.327249 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.327220 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-token" pod "f67c14bb-f110-45db-b9d8-4800a84b7277" (UID: "f67c14bb-f110-45db-b9d8-4800a84b7277"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:52.327249 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.327227 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-envoy" pod "f67c14bb-f110-45db-b9d8-4800a84b7277" (UID: "f67c14bb-f110-45db-b9d8-4800a84b7277"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.327841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.327820 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-podinfo" pod "f67c14bb-f110-45db-b9d8-4800a84b7277" (UID: "f67c14bb-f110-45db-b9d8-4800a84b7277"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:15:52.327940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.327924 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-kube-api-access-9g8x7" pod "f67c14bb-f110-45db-b9d8-4800a84b7277" (UID: "f67c14bb-f110-45db-b9d8-4800a84b7277"). InnerVolumeSpecName "kube-api-access-9g8x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:52.427248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.427210 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.427248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.427241 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.427248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.427253 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g8x7\" (UniqueName: \"kubernetes.io/projected/f67c14bb-f110-45db-b9d8-4800a84b7277-kube-api-access-9g8x7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.427539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.427266 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67c14bb-f110-45db-b9d8-4800a84b7277-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.813290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.813254 5001 generic.go:356] "Generic (PLEG): container finished" podID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerID="cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d" exitCode=0 Oct 30 06:15:52.813505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.813328 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" Oct 30 06:15:52.813505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.813344 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" event={"ID":"f67c14bb-f110-45db-b9d8-4800a84b7277","Type":"ContainerDied","Data":"cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d"} Oct 30 06:15:52.813505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.813402 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl" event={"ID":"f67c14bb-f110-45db-b9d8-4800a84b7277","Type":"ContainerDied","Data":"85ebe40318517ed8856608b17249d8070a43eb75c4dd403eae35a30d2ca94af4"} Oct 30 06:15:52.813505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.813429 5001 scope.go:122] "RemoveContainer" containerID="cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d" Oct 30 06:15:52.825985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.825962 5001 scope.go:122] "RemoveContainer" containerID="f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52" Oct 30 06:15:52.937113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.936474 5001 scope.go:122] "RemoveContainer" containerID="cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d" Oct 30 06:15:52.953044 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:52.952490 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d\": container with ID starting with cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d not found: ID does not exist" containerID="cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d" Oct 30 06:15:52.953044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.952544 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d"} err="failed to get container status \"cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d\": rpc error: code = NotFound desc = could not find container \"cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d\": container with ID starting with cc7de108cf11f500416fbed7b8b9f139e692424643f9ef8c3265cc25cb36906d not found: ID does not exist" Oct 30 06:15:52.953044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.952574 5001 scope.go:122] "RemoveContainer" containerID="f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52" Oct 30 06:15:52.973330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.959852 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl"] Oct 30 06:15:52.973330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.972948 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-kx4kl"] Oct 30 06:15:52.973330 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:15:52.973198 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52\": container with ID starting with f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52 not found: ID does not exist" containerID="f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52" Oct 30 06:15:52.973330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:52.973251 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52"} err="failed to get container status \"f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52\": rpc error: code = NotFound desc = could not find container \"f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52\": container with ID starting with f0cea8de6d55b005c067c6d100dee509b722c26a96080a8d82ece32254859d52 not found: ID does not exist" Oct 30 06:15:54.118895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:54.118864 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" path="/var/lib/kubelet/pods/f67c14bb-f110-45db-b9d8-4800a84b7277/volumes" Oct 30 06:15:57.012771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.012739 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn"] Oct 30 06:15:57.025427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.025398 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba44b248-dbsqn"] Oct 30 06:15:57.292843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.292727 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7"] Oct 30 06:15:57.293421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.293404 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" Oct 30 06:15:57.293421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.293420 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" Oct 30 06:15:57.293592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.293464 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f04d0288-0153-4399-bf11-0a137ae83450" containerName="revoke-keys" Oct 30 06:15:57.293592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.293469 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04d0288-0153-4399-bf11-0a137ae83450" containerName="revoke-keys" Oct 30 06:15:57.293592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.293566 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04d0288-0153-4399-bf11-0a137ae83450" containerName="revoke-keys" Oct 30 06:15:57.293592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.293578 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" Oct 30 06:15:57.293592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.293586 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" Oct 30 06:15:57.297265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.297244 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:15:57.318989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.318957 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7"] Oct 30 06:15:57.325066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.325041 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:15:57.325400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.325341 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:15:57.325400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.325350 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:15:57.325795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.325773 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:15:57.402319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.402272 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:15:57.402497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.402423 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:15:57.402497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.402482 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl69b\" (UniqueName: \"kubernetes.io/projected/ebca1506-e523-4b3f-81d1-e40432e5d3df-kube-api-access-bl69b\") pod \"maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:15:57.504138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.504100 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:15:57.504358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.504182 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:15:57.504358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.504220 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl69b\" (UniqueName: \"kubernetes.io/projected/ebca1506-e523-4b3f-81d1-e40432e5d3df-kube-api-access-bl69b\") pod \"maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:15:57.504798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.504780 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:15:57.507085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.507051 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:15:57.528252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.528226 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl69b\" (UniqueName: \"kubernetes.io/projected/ebca1506-e523-4b3f-81d1-e40432e5d3df-kube-api-access-bl69b\") pod \"maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:15:57.614433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.614401 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:15:57.933414 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:15:57.933384 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebca1506_e523_4b3f_81d1_e40432e5d3df.slice/crio-a8890ccc802c69b1008e3e1b791d5d95502b12f308dd4c300167c80bf298c6b2 WatchSource:0}: Error finding container a8890ccc802c69b1008e3e1b791d5d95502b12f308dd4c300167c80bf298c6b2: Status 404 returned error can't find the container with id a8890ccc802c69b1008e3e1b791d5d95502b12f308dd4c300167c80bf298c6b2 Oct 30 06:15:57.933556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.933533 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7"] Oct 30 06:15:57.969787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:57.969556 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff"] Oct 30 06:15:58.002459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:58.002417 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s"] Oct 30 06:15:58.121066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:58.121033 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd5a5be-37ed-437a-b707-d3ae59c793af" path="/var/lib/kubelet/pods/acd5a5be-37ed-437a-b707-d3ae59c793af/volumes" Oct 30 06:15:58.909716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:58.909677 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" event={"ID":"ebca1506-e523-4b3f-81d1-e40432e5d3df","Type":"ContainerStarted","Data":"9b8301478f47b1219462d509b7813d2f5cfe1f2ce395bb376a6d1571b7d882c6"} Oct 30 06:15:58.909905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:58.909751 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" event={"ID":"ebca1506-e523-4b3f-81d1-e40432e5d3df","Type":"ContainerStarted","Data":"a8890ccc802c69b1008e3e1b791d5d95502b12f308dd4c300167c80bf298c6b2"} Oct 30 06:15:58.960376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:58.959684 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" podStartSLOduration=1.959663347 podStartE2EDuration="1.959663347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:58.951776315 +0000 UTC m=+1372.947733339" watchObservedRunningTime="2026-10-30 06:15:58.959663347 +0000 UTC m=+1372.955620361" Oct 30 06:15:59.919512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:59.919480 5001 generic.go:356] "Generic (PLEG): container finished" podID="ebca1506-e523-4b3f-81d1-e40432e5d3df" containerID="9b8301478f47b1219462d509b7813d2f5cfe1f2ce395bb376a6d1571b7d882c6" exitCode=0 Oct 30 06:15:59.919880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:15:59.919520 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" event={"ID":"ebca1506-e523-4b3f-81d1-e40432e5d3df","Type":"ContainerDied","Data":"9b8301478f47b1219462d509b7813d2f5cfe1f2ce395bb376a6d1571b7d882c6"} Oct 30 06:16:01.141838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.141809 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:16:01.202391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.202266 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebca1506-e523-4b3f-81d1-e40432e5d3df-kube-api-access-bl69b\" (UniqueName: \"kubernetes.io/projected/ebca1506-e523-4b3f-81d1-e40432e5d3df-kube-api-access-bl69b\") pod \"ebca1506-e523-4b3f-81d1-e40432e5d3df\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " Oct 30 06:16:01.202391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.202339 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-openshift-ca\") pod \"ebca1506-e523-4b3f-81d1-e40432e5d3df\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " Oct 30 06:16:01.202631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.202554 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-xks-ca\") pod \"ebca1506-e523-4b3f-81d1-e40432e5d3df\" (UID: \"ebca1506-e523-4b3f-81d1-e40432e5d3df\") " Oct 30 06:16:01.203080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.203049 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-openshift-ca" pod "ebca1506-e523-4b3f-81d1-e40432e5d3df" (UID: "ebca1506-e523-4b3f-81d1-e40432e5d3df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:01.206464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.205647 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-xks-ca" pod "ebca1506-e523-4b3f-81d1-e40432e5d3df" (UID: "ebca1506-e523-4b3f-81d1-e40432e5d3df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:01.216687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.216660 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebca1506-e523-4b3f-81d1-e40432e5d3df-kube-api-access-bl69b" pod "ebca1506-e523-4b3f-81d1-e40432e5d3df" (UID: "ebca1506-e523-4b3f-81d1-e40432e5d3df"). InnerVolumeSpecName "kube-api-access-bl69b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:01.304361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.304326 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:01.304361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.304357 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl69b\" (UniqueName: \"kubernetes.io/projected/ebca1506-e523-4b3f-81d1-e40432e5d3df-kube-api-access-bl69b\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:01.304361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.304370 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebca1506-e523-4b3f-81d1-e40432e5d3df-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:01.960806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.960773 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" Oct 30 06:16:01.961024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.960814 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7" event={"ID":"ebca1506-e523-4b3f-81d1-e40432e5d3df","Type":"ContainerDied","Data":"a8890ccc802c69b1008e3e1b791d5d95502b12f308dd4c300167c80bf298c6b2"} Oct 30 06:16:01.961024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:01.960875 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8890ccc802c69b1008e3e1b791d5d95502b12f308dd4c300167c80bf298c6b2" Oct 30 06:16:02.041103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:02.041069 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f"] Oct 30 06:16:02.046034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:02.046006 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e188c1ae-wbl5f"] Oct 30 06:16:02.118990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:02.118956 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f315c894-1fe5-410f-ba57-0e0d24feb024" path="/var/lib/kubelet/pods/f315c894-1fe5-410f-ba57-0e0d24feb024/volumes" Oct 30 06:16:02.969936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:02.969865 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" podUID="f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0" containerName="payload-processing" containerID="cri-o://92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72" gracePeriod=25 Oct 30 06:16:03.002898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.002854 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" podUID="8da8e900-94fb-4359-b1d9-25dd84b75281" containerName="payload-pre-processing" containerID="cri-o://05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc" gracePeriod=25 Oct 30 06:16:03.426468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.426431 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:16:03.510171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.476623 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:16:03.594382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.574958 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-kube-api-access-q69xf\" (UniqueName: \"kubernetes.io/projected/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-kube-api-access-q69xf\") pod \"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0\" (UID: \"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0\") " Oct 30 06:16:03.595815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.595786 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-plugins-config-volume\") pod \"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0\" (UID: \"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0\") " Oct 30 06:16:03.608548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.596408 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-plugins-config-volume" pod "f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0" (UID: "f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:03.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.631937 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-kube-api-access-q69xf" pod "f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0" (UID: "f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0"). InnerVolumeSpecName "kube-api-access-q69xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:03.677570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.676463 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:16:03.699022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.698978 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q69xf\" (UniqueName: \"kubernetes.io/projected/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-kube-api-access-q69xf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:03.699022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.699023 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:03.800354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.800321 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da8e900-94fb-4359-b1d9-25dd84b75281-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da8e900-94fb-4359-b1d9-25dd84b75281-plugins-config-volume\") pod \"8da8e900-94fb-4359-b1d9-25dd84b75281\" (UID: \"8da8e900-94fb-4359-b1d9-25dd84b75281\") " Oct 30 06:16:03.800586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.800507 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da8e900-94fb-4359-b1d9-25dd84b75281-kube-api-access-d9kc5\" (UniqueName: \"kubernetes.io/projected/8da8e900-94fb-4359-b1d9-25dd84b75281-kube-api-access-d9kc5\") pod \"8da8e900-94fb-4359-b1d9-25dd84b75281\" (UID: \"8da8e900-94fb-4359-b1d9-25dd84b75281\") " Oct 30 06:16:03.800775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.800746 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da8e900-94fb-4359-b1d9-25dd84b75281-plugins-config-volume" pod "8da8e900-94fb-4359-b1d9-25dd84b75281" (UID: "8da8e900-94fb-4359-b1d9-25dd84b75281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:03.800977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.800960 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da8e900-94fb-4359-b1d9-25dd84b75281-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:03.802850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.802821 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da8e900-94fb-4359-b1d9-25dd84b75281-kube-api-access-d9kc5" pod "8da8e900-94fb-4359-b1d9-25dd84b75281" (UID: "8da8e900-94fb-4359-b1d9-25dd84b75281"). InnerVolumeSpecName "kube-api-access-d9kc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:03.902586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.902544 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9kc5\" (UniqueName: \"kubernetes.io/projected/8da8e900-94fb-4359-b1d9-25dd84b75281-kube-api-access-d9kc5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:03.990070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.988189 5001 generic.go:356] "Generic (PLEG): container finished" podID="f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0" containerID="92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72" exitCode=0 Oct 30 06:16:03.990070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.988244 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" Oct 30 06:16:03.990070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.988246 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" event={"ID":"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0","Type":"ContainerDied","Data":"92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72"} Oct 30 06:16:03.990070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.988334 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff" event={"ID":"f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0","Type":"ContainerDied","Data":"57c6365476b097dffbbd205b9d76948123ac81f1d597c49bfc3449fbec689619"} Oct 30 06:16:03.990070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.988367 5001 scope.go:122] "RemoveContainer" containerID="92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72" Oct 30 06:16:03.992014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.991987 5001 generic.go:356] "Generic (PLEG): container finished" podID="8da8e900-94fb-4359-b1d9-25dd84b75281" containerID="05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc" exitCode=0 Oct 30 06:16:03.992217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.992192 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" event={"ID":"8da8e900-94fb-4359-b1d9-25dd84b75281","Type":"ContainerDied","Data":"05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc"} Oct 30 06:16:03.992344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.992246 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" event={"ID":"8da8e900-94fb-4359-b1d9-25dd84b75281","Type":"ContainerDied","Data":"a1a940b262c78257b6330226062c5082639134c29e3b8ba04619fe885ec33a92"} Oct 30 06:16:03.992443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:03.992346 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s" Oct 30 06:16:04.011463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:04.011430 5001 scope.go:122] "RemoveContainer" containerID="92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72" Oct 30 06:16:04.108060 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:16:04.107989 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72\": container with ID starting with 92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72 not found: ID does not exist" containerID="92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72" Oct 30 06:16:04.116563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:04.116435 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72"} err="failed to get container status \"92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72\": rpc error: code = NotFound desc = could not find container \"92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72\": container with ID starting with 92dd8a81ed9e45b902a50b4fbae46372aad7e01a0d9ec7c3cac426854ee47f72 not found: ID does not exist" Oct 30 06:16:04.116563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:04.116509 5001 scope.go:122] "RemoveContainer" containerID="05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc" Oct 30 06:16:04.277535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:04.265077 5001 scope.go:122] "RemoveContainer" containerID="05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc" Oct 30 06:16:04.277535 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:16:04.267629 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc\": container with ID starting with 05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc not found: ID does not exist" containerID="05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc" Oct 30 06:16:04.277535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:04.267687 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc"} err="failed to get container status \"05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc\": rpc error: code = NotFound desc = could not find container \"05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc\": container with ID starting with 05c7b59f7cfadcb83d10e3e5c6f619eb56913ed6a71a6358def8524870477fcc not found: ID does not exist" Oct 30 06:16:04.334216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:04.334177 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff"] Oct 30 06:16:04.336242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:04.336210 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe108e-6d695b9794-ftnff"] Oct 30 06:16:04.344121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:04.344090 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s"] Oct 30 06:16:04.349878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:04.349850 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe108e-b4fcd4c55-q685s"] Oct 30 06:16:06.041005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:06.040971 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2"] Oct 30 06:16:06.065333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:06.060397 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7305ea32-jrjp2"] Oct 30 06:16:06.119615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:06.119584 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c8170d-bb38-41a4-be87-2377cfdefb9d" path="/var/lib/kubelet/pods/24c8170d-bb38-41a4-be87-2377cfdefb9d/volumes" Oct 30 06:16:06.119994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:06.119980 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da8e900-94fb-4359-b1d9-25dd84b75281" path="/var/lib/kubelet/pods/8da8e900-94fb-4359-b1d9-25dd84b75281/volumes" Oct 30 06:16:06.120388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:06.120374 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0" path="/var/lib/kubelet/pods/f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0/volumes" Oct 30 06:16:07.042878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:07.042849 5001 scope.go:122] "RemoveContainer" containerID="a28c9cf08a9c603d73d33f5cf28d6311e4fb06624c2c1c8041d585ca7b0fbc0f" Oct 30 06:16:07.070533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:07.068473 5001 scope.go:122] "RemoveContainer" containerID="c63bb32308c7cb9f3ffd6b7a63a7f99a854deecf880a4c6c7cbd3d4fd3e24015" Oct 30 06:16:08.286855 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:16:08.286796 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.627418 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6"] Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.628504 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0" containerName="payload-processing" Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.628526 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0" containerName="payload-processing" Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.628555 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8da8e900-94fb-4359-b1d9-25dd84b75281" containerName="payload-pre-processing" Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.628593 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da8e900-94fb-4359-b1d9-25dd84b75281" containerName="payload-pre-processing" Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.628639 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.628647 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67c14bb-f110-45db-b9d8-4800a84b7277" containerName="istio-proxy" Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.628663 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ebca1506-e523-4b3f-81d1-e40432e5d3df" containerName="revoke-keys" Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.628680 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebca1506-e523-4b3f-81d1-e40432e5d3df" containerName="revoke-keys" Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.628941 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f4ebc2-6a75-4dd3-a1c8-42b4dd900bc0" containerName="payload-processing" Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.628959 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da8e900-94fb-4359-b1d9-25dd84b75281" containerName="payload-pre-processing" Oct 30 06:16:08.634331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.628978 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebca1506-e523-4b3f-81d1-e40432e5d3df" containerName="revoke-keys" Oct 30 06:16:08.688333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.686876 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6"] Oct 30 06:16:08.688333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.687007 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:08.738384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.707439 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:16:08.738384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.722632 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:16:08.738384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.722724 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:16:08.808330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.804631 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:16:08.832416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.831104 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba69e6d-xz6t6\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:08.832416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.831181 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k984\" (UniqueName: \"kubernetes.io/projected/0cf05d5f-d505-4701-8381-ace1143a93b2-kube-api-access-9k984\") pod \"maas-api-revoke-sub-eba69e6d-xz6t6\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:08.839422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.831378 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba69e6d-xz6t6\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:08.941464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.941366 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba69e6d-xz6t6\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:08.941464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.941411 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k984\" (UniqueName: \"kubernetes.io/projected/0cf05d5f-d505-4701-8381-ace1143a93b2-kube-api-access-9k984\") pod \"maas-api-revoke-sub-eba69e6d-xz6t6\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:08.941727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.941491 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba69e6d-xz6t6\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:08.942356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.942246 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba69e6d-xz6t6\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:08.944236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.944205 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba69e6d-xz6t6\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:08.979483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:08.979449 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k984\" (UniqueName: \"kubernetes.io/projected/0cf05d5f-d505-4701-8381-ace1143a93b2-kube-api-access-9k984\") pod \"maas-api-revoke-sub-eba69e6d-xz6t6\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:09.077017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:09.076547 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:09.393631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:09.393244 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6"] Oct 30 06:16:10.145253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:10.145151 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" event={"ID":"0cf05d5f-d505-4701-8381-ace1143a93b2","Type":"ContainerStarted","Data":"82374251e9b5ebb1a23cd2225a81bf478381fc1f5156b5227fc75a1417aea781"} Oct 30 06:16:10.145253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:10.145217 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" event={"ID":"0cf05d5f-d505-4701-8381-ace1143a93b2","Type":"ContainerStarted","Data":"4520c575507364b4d02962071be9cb5a02c05b3a15b359176929e8af12262da1"} Oct 30 06:16:10.193952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:10.193896 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" podStartSLOduration=2.193880631 podStartE2EDuration="2.193880631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:10.190190527 +0000 UTC m=+1384.186147539" watchObservedRunningTime="2026-10-30 06:16:10.193880631 +0000 UTC m=+1384.189837642" Oct 30 06:16:11.155526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:11.155494 5001 generic.go:356] "Generic (PLEG): container finished" podID="0cf05d5f-d505-4701-8381-ace1143a93b2" containerID="82374251e9b5ebb1a23cd2225a81bf478381fc1f5156b5227fc75a1417aea781" exitCode=0 Oct 30 06:16:11.156019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:11.155579 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" event={"ID":"0cf05d5f-d505-4701-8381-ace1143a93b2","Type":"ContainerDied","Data":"82374251e9b5ebb1a23cd2225a81bf478381fc1f5156b5227fc75a1417aea781"} Oct 30 06:16:12.393913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:12.393520 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:12.451931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:12.451828 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-xks-ca\") pod \"0cf05d5f-d505-4701-8381-ace1143a93b2\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " Oct 30 06:16:12.451931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:12.451898 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-openshift-ca\") pod \"0cf05d5f-d505-4701-8381-ace1143a93b2\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " Oct 30 06:16:12.452187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:12.452039 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf05d5f-d505-4701-8381-ace1143a93b2-kube-api-access-9k984\" (UniqueName: \"kubernetes.io/projected/0cf05d5f-d505-4701-8381-ace1143a93b2-kube-api-access-9k984\") pod \"0cf05d5f-d505-4701-8381-ace1143a93b2\" (UID: \"0cf05d5f-d505-4701-8381-ace1143a93b2\") " Oct 30 06:16:12.452782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:12.452712 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-openshift-ca" pod "0cf05d5f-d505-4701-8381-ace1143a93b2" (UID: "0cf05d5f-d505-4701-8381-ace1143a93b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:12.461331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:12.461185 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:12.466951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:12.466897 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf05d5f-d505-4701-8381-ace1143a93b2-kube-api-access-9k984" pod "0cf05d5f-d505-4701-8381-ace1143a93b2" (UID: "0cf05d5f-d505-4701-8381-ace1143a93b2"). InnerVolumeSpecName "kube-api-access-9k984". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:12.469420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:12.469374 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-xks-ca" pod "0cf05d5f-d505-4701-8381-ace1143a93b2" (UID: "0cf05d5f-d505-4701-8381-ace1143a93b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:12.563377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:12.563339 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf05d5f-d505-4701-8381-ace1143a93b2-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:12.563377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:12.563376 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k984\" (UniqueName: \"kubernetes.io/projected/0cf05d5f-d505-4701-8381-ace1143a93b2-kube-api-access-9k984\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:13.194841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:13.194721 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" Oct 30 06:16:13.194841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:13.194730 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6" event={"ID":"0cf05d5f-d505-4701-8381-ace1143a93b2","Type":"ContainerDied","Data":"4520c575507364b4d02962071be9cb5a02c05b3a15b359176929e8af12262da1"} Oct 30 06:16:13.194841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:13.194781 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4520c575507364b4d02962071be9cb5a02c05b3a15b359176929e8af12262da1" Oct 30 06:16:14.742453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:14.742417 5001 patch_prober.go:29] interesting pod/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:16:14.742970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:14.742527 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:16:16.045538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:16.045502 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6"] Oct 30 06:16:16.060684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:16.060631 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70652026-m5gb6"] Oct 30 06:16:16.117816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:16.117783 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4094479-4ea6-4004-9ae7-06ab553974c3" path="/var/lib/kubelet/pods/a4094479-4ea6-4004-9ae7-06ab553974c3/volumes" Oct 30 06:16:16.224710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:16.224678 5001 generic.go:356] "Generic (PLEG): container finished" podID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerID="db541d13c4fdd3522aec928d0985373f7374bf475fa416591adb849b12897636" exitCode=0 Oct 30 06:16:16.224904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:16.224747 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" event={"ID":"bf6842a8-36df-4ea9-8639-a7a7cb18e577","Type":"ContainerDied","Data":"db541d13c4fdd3522aec928d0985373f7374bf475fa416591adb849b12897636"} Oct 30 06:16:16.225231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:16.225213 5001 scope.go:122] "RemoveContainer" containerID="db541d13c4fdd3522aec928d0985373f7374bf475fa416591adb849b12897636" Oct 30 06:16:16.226813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:16.226780 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:16.226894 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:16.226853 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:16.226894 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:16.226887 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:17.236982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:17.236899 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" event={"ID":"bf6842a8-36df-4ea9-8639-a7a7cb18e577","Type":"ContainerStarted","Data":"6e62ce5bdcd3faefa71532b83323531c5d0d9251562db1a2df7cf641e4e919ae"} Oct 30 06:16:18.121534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:18.121502 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:16:18.121719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:18.121688 5001 patch_prober.go:29] interesting pod/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 06:16:18.121779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:18.121756 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 06:16:19.122125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:19.122091 5001 patch_prober.go:29] interesting pod/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 06:16:19.122548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:19.122147 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 06:16:20.123430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:20.123394 5001 patch_prober.go:29] interesting pod/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 06:16:20.124072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:20.124040 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 06:16:21.056461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:21.056374 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k"] Oct 30 06:16:21.100712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:21.100673 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b64b77-87p2k"] Oct 30 06:16:21.124746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:21.124710 5001 patch_prober.go:29] interesting pod/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 06:16:21.125723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:21.125293 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 06:16:22.128172 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:22.127662 5001 patch_prober.go:29] interesting pod/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 06:16:22.128172 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:22.127738 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 06:16:22.166333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:22.165709 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12494765-b808-4020-af35-349e1b46fa46" path="/var/lib/kubelet/pods/12494765-b808-4020-af35-349e1b46fa46/volumes" Oct 30 06:16:22.647328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:22.646423 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:16:22.752454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:22.752400 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:16:22.785579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:22.785521 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:16:22.839465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:22.839423 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2ltk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:16:22.839788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:22.839755 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:16:23.126407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:23.126370 5001 patch_prober.go:29] interesting pod/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 06:16:23.126715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:23.126684 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 06:16:24.210124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:24.210088 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:16:24.212533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:24.212508 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:16:24.214232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:24.214205 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:16:26.486336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:26.484627 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk"] Oct 30 06:16:26.505050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:26.505012 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe108e-c25nk"] Oct 30 06:16:28.119343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:28.119254 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04d0288-0153-4399-bf11-0a137ae83450" path="/var/lib/kubelet/pods/f04d0288-0153-4399-bf11-0a137ae83450/volumes" Oct 30 06:16:31.046464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:31.046431 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v"] Oct 30 06:16:31.060787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:31.060750 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efed0ff1-n5m2v"] Oct 30 06:16:32.048184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:32.048143 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c"] Oct 30 06:16:32.078732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:32.078573 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fda812d-hzv7c"] Oct 30 06:16:32.121505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:32.121477 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0514ee-2b01-4075-b812-c3680de46879" path="/var/lib/kubelet/pods/2d0514ee-2b01-4075-b812-c3680de46879/volumes" Oct 30 06:16:32.122021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:32.122007 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c322a542-c1b4-4ba9-8ef6-58d062b9a7a1" path="/var/lib/kubelet/pods/c322a542-c1b4-4ba9-8ef6-58d062b9a7a1/volumes" Oct 30 06:16:34.141157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:34.141124 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7"] Oct 30 06:16:34.141937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:34.141910 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c15d3a-r5cl7"] Oct 30 06:16:35.063614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:35.063574 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b"] Oct 30 06:16:35.064059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:35.064027 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" containerID="cri-o://a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964" gracePeriod=30 Oct 30 06:16:36.143091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:36.143044 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebca1506-e523-4b3f-81d1-e40432e5d3df" path="/var/lib/kubelet/pods/ebca1506-e523-4b3f-81d1-e40432e5d3df/volumes" Oct 30 06:16:36.624044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:36.623998 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:16:38.289633 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:16:38.289552 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:16:40.200520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.191590 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc"] Oct 30 06:16:40.200520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.191641 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c07a1f2-skfmc"] Oct 30 06:16:40.517283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.517254 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:16:40.712826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.712730 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-podinfo\") pod \"8cfbab7a-23c7-4712-b0c6-65529f21890e\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " Oct 30 06:16:40.712826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.712794 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-token\" (UniqueName: \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-token\") pod \"8cfbab7a-23c7-4712-b0c6-65529f21890e\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " Oct 30 06:16:40.713065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.712836 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-envoy\") pod \"8cfbab7a-23c7-4712-b0c6-65529f21890e\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " Oct 30 06:16:40.713065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.712863 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-kube-api-access-9jl4b\" (UniqueName: \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-kube-api-access-9jl4b\") pod \"8cfbab7a-23c7-4712-b0c6-65529f21890e\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " Oct 30 06:16:40.713065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.712912 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-certs\") pod \"8cfbab7a-23c7-4712-b0c6-65529f21890e\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " Oct 30 06:16:40.713218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.713200 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-socket\") pod \"8cfbab7a-23c7-4712-b0c6-65529f21890e\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " Oct 30 06:16:40.713360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.713278 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-certs" pod "8cfbab7a-23c7-4712-b0c6-65529f21890e" (UID: "8cfbab7a-23c7-4712-b0c6-65529f21890e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:40.713537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.713459 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfbab7a-23c7-4712-b0c6-65529f21890e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cfbab7a-23c7-4712-b0c6-65529f21890e-istiod-ca-cert\") pod \"8cfbab7a-23c7-4712-b0c6-65529f21890e\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " Oct 30 06:16:40.713537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.713478 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-socket" pod "8cfbab7a-23c7-4712-b0c6-65529f21890e" (UID: "8cfbab7a-23c7-4712-b0c6-65529f21890e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:40.713537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.713514 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-data\") pod \"8cfbab7a-23c7-4712-b0c6-65529f21890e\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " Oct 30 06:16:40.713847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.713573 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-credential-socket\") pod \"8cfbab7a-23c7-4712-b0c6-65529f21890e\" (UID: \"8cfbab7a-23c7-4712-b0c6-65529f21890e\") " Oct 30 06:16:40.713991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.713926 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-data" pod "8cfbab7a-23c7-4712-b0c6-65529f21890e" (UID: "8cfbab7a-23c7-4712-b0c6-65529f21890e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:40.714127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.714073 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-credential-socket" pod "8cfbab7a-23c7-4712-b0c6-65529f21890e" (UID: "8cfbab7a-23c7-4712-b0c6-65529f21890e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:40.714127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.714080 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfbab7a-23c7-4712-b0c6-65529f21890e-istiod-ca-cert" pod "8cfbab7a-23c7-4712-b0c6-65529f21890e" (UID: "8cfbab7a-23c7-4712-b0c6-65529f21890e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:40.714512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.714486 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:40.714512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.714510 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:40.714628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.714523 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:40.714628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.714535 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cfbab7a-23c7-4712-b0c6-65529f21890e-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:40.714628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.714548 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:40.715369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.715345 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-podinfo" pod "8cfbab7a-23c7-4712-b0c6-65529f21890e" (UID: "8cfbab7a-23c7-4712-b0c6-65529f21890e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:16:40.715472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.715384 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-envoy" pod "8cfbab7a-23c7-4712-b0c6-65529f21890e" (UID: "8cfbab7a-23c7-4712-b0c6-65529f21890e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:40.715774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.715746 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-token" pod "8cfbab7a-23c7-4712-b0c6-65529f21890e" (UID: "8cfbab7a-23c7-4712-b0c6-65529f21890e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:40.715774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.715761 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-kube-api-access-9jl4b" pod "8cfbab7a-23c7-4712-b0c6-65529f21890e" (UID: "8cfbab7a-23c7-4712-b0c6-65529f21890e"). InnerVolumeSpecName "kube-api-access-9jl4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:40.759206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.759173 5001 generic.go:356] "Generic (PLEG): container finished" podID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerID="a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964" exitCode=0 Oct 30 06:16:40.759436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.759228 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" Oct 30 06:16:40.759436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.759240 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" event={"ID":"8cfbab7a-23c7-4712-b0c6-65529f21890e","Type":"ContainerDied","Data":"a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964"} Oct 30 06:16:40.759436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.759293 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b" event={"ID":"8cfbab7a-23c7-4712-b0c6-65529f21890e","Type":"ContainerDied","Data":"9ee60c8fbca4ae1ea567334aa4e68521fb7d246f1f08122d6259a0f968634be6"} Oct 30 06:16:40.759436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.759340 5001 scope.go:122] "RemoveContainer" containerID="a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964" Oct 30 06:16:40.774617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.774591 5001 scope.go:122] "RemoveContainer" containerID="1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253" Oct 30 06:16:40.788385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.788359 5001 scope.go:122] "RemoveContainer" containerID="a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964" Oct 30 06:16:40.788763 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:16:40.788740 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964\": container with ID starting with a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964 not found: ID does not exist" containerID="a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964" Oct 30 06:16:40.788889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.788776 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964"} err="failed to get container status \"a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964\": rpc error: code = NotFound desc = could not find container \"a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964\": container with ID starting with a16dcc2034bf96fc1871f11c842a3b82a70b11867613cc77e4975675b7e5f964 not found: ID does not exist" Oct 30 06:16:40.788889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.788803 5001 scope.go:122] "RemoveContainer" containerID="1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253" Oct 30 06:16:40.789113 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:16:40.789095 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253\": container with ID starting with 1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253 not found: ID does not exist" containerID="1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253" Oct 30 06:16:40.789173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.789122 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253"} err="failed to get container status \"1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253\": rpc error: code = NotFound desc = could not find container \"1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253\": container with ID starting with 1b788583766883251e0d41ac7daede7de6940fedf98cc1598df06e70be15c253 not found: ID does not exist" Oct 30 06:16:40.816780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.816740 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:40.816780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.816775 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:40.816780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.816789 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cfbab7a-23c7-4712-b0c6-65529f21890e-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:40.817123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.816802 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jl4b\" (UniqueName: \"kubernetes.io/projected/8cfbab7a-23c7-4712-b0c6-65529f21890e-kube-api-access-9jl4b\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:40.822092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.822058 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b"] Oct 30 06:16:40.826360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:40.826328 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-fe108e-openshift-default-6487d559b9-6ff5b"] Oct 30 06:16:42.073235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.071514 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn"] Oct 30 06:16:42.073235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.071882 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" containerID="cri-o://b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d" gracePeriod=30 Oct 30 06:16:42.130878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.130831 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a0f834-6541-4317-9e9b-ffd9b0653e22" path="/var/lib/kubelet/pods/34a0f834-6541-4317-9e9b-ffd9b0653e22/volumes" Oct 30 06:16:42.131431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.131409 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" path="/var/lib/kubelet/pods/8cfbab7a-23c7-4712-b0c6-65529f21890e/volumes" Oct 30 06:16:42.144395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.144367 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl"] Oct 30 06:16:42.145136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.145113 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" Oct 30 06:16:42.145136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.145130 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" Oct 30 06:16:42.145239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.145161 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0cf05d5f-d505-4701-8381-ace1143a93b2" containerName="revoke-keys" Oct 30 06:16:42.145239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.145170 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf05d5f-d505-4701-8381-ace1143a93b2" containerName="revoke-keys" Oct 30 06:16:42.145320 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.145295 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" Oct 30 06:16:42.145360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.145326 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf05d5f-d505-4701-8381-ace1143a93b2" containerName="revoke-keys" Oct 30 06:16:42.145360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.145345 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" Oct 30 06:16:42.150092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.150073 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.171956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.171926 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-33c99c-openshift-default-dockercfg-bcqfx\"" Oct 30 06:16:42.186589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.186557 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl"] Oct 30 06:16:42.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.242609 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-socket\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.242760 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-credential-socket\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.242832 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c3319d-5388-4262-8abc-3bd921947b04-istio-podinfo\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.242880 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4cg\" (UniqueName: \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-kube-api-access-pj4cg\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.242953 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-certs\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.242984 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-envoy\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.243067 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-istio-token\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.243096 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c3319d-5388-4262-8abc-3bd921947b04-istiod-ca-cert\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.245332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.243139 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-data\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.367388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.358916 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-certs\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.367388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.358977 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-envoy\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.367388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.359076 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-istio-token\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.367388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.359105 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c3319d-5388-4262-8abc-3bd921947b04-istiod-ca-cert\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.367388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.359143 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-data\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.367388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.359265 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-socket\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.367388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.359366 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-credential-socket\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.367388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.359448 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c3319d-5388-4262-8abc-3bd921947b04-istio-podinfo\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.367388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.359505 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4cg\" (UniqueName: \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-kube-api-access-pj4cg\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.405331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.403144 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c3319d-5388-4262-8abc-3bd921947b04-istiod-ca-cert\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.405331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.403438 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-certs\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.423237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.422268 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-socket\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.423237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.422569 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-data\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.423237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.422796 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-credential-socket\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.469691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.469651 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c3319d-5388-4262-8abc-3bd921947b04-istio-podinfo\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.483438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.483408 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-istio-token\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.499349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.497183 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-envoy\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.500251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.500189 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4cg\" (UniqueName: \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-kube-api-access-pj4cg\") pod \"e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.766384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.765549 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:42.793573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.793542 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p"] Oct 30 06:16:42.794230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.794202 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" Oct 30 06:16:42.794230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.794216 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfbab7a-23c7-4712-b0c6-65529f21890e" containerName="istio-proxy" Oct 30 06:16:42.798173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.798152 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:42.880331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.879852 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:16:42.899454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.896747 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:16:42.899454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.897455 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:16:42.899454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.897535 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:16:42.910533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.908879 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7a606-nf55p\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:42.910533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.909095 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7a606-nf55p\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:42.910533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.909436 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwwks\" (UniqueName: \"kubernetes.io/projected/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-kube-api-access-dwwks\") pod \"maas-api-revoke-keys-e2e-api-b-f7a606-nf55p\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:42.911223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:42.911149 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p"] Oct 30 06:16:43.011004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.010965 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwwks\" (UniqueName: \"kubernetes.io/projected/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-kube-api-access-dwwks\") pod \"maas-api-revoke-keys-e2e-api-b-f7a606-nf55p\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:43.011208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.011055 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7a606-nf55p\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:43.011282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.011204 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7a606-nf55p\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:43.011743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.011716 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7a606-nf55p\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:43.013994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.013971 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7a606-nf55p\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:43.035702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.035630 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwwks\" (UniqueName: \"kubernetes.io/projected/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-kube-api-access-dwwks\") pod \"maas-api-revoke-keys-e2e-api-b-f7a606-nf55p\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:43.105207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.105174 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl"] Oct 30 06:16:43.106257 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:16:43.106224 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c3319d_5388_4262_8abc_3bd921947b04.slice/crio-eef16cd1a8025342e7f01a92b959a19137e0a395c6ce04dbe7c709cdcd1e9f27 WatchSource:0}: Error finding container eef16cd1a8025342e7f01a92b959a19137e0a395c6ce04dbe7c709cdcd1e9f27: Status 404 returned error can't find the container with id eef16cd1a8025342e7f01a92b959a19137e0a395c6ce04dbe7c709cdcd1e9f27 Oct 30 06:16:43.108968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.108937 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:43.109048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.109003 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:43.109048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.109033 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:43.129636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.129610 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:43.450607 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:16:43.450581 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdea9c7b_6f75_46ba_a1f6_9771c6358d4d.slice/crio-30d28aa3dd31c589181a67221cf712b2bdd3c0d004085b26e6503f4dea9c9c95 WatchSource:0}: Error finding container 30d28aa3dd31c589181a67221cf712b2bdd3c0d004085b26e6503f4dea9c9c95: Status 404 returned error can't find the container with id 30d28aa3dd31c589181a67221cf712b2bdd3c0d004085b26e6503f4dea9c9c95 Oct 30 06:16:43.452911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.452860 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p"] Oct 30 06:16:43.793358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.793242 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" event={"ID":"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d","Type":"ContainerStarted","Data":"30d28aa3dd31c589181a67221cf712b2bdd3c0d004085b26e6503f4dea9c9c95"} Oct 30 06:16:43.795230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.795197 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" event={"ID":"20c3319d-5388-4262-8abc-3bd921947b04","Type":"ContainerStarted","Data":"a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71"} Oct 30 06:16:43.795375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.795253 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" event={"ID":"20c3319d-5388-4262-8abc-3bd921947b04","Type":"ContainerStarted","Data":"eef16cd1a8025342e7f01a92b959a19137e0a395c6ce04dbe7c709cdcd1e9f27"} Oct 30 06:16:43.857962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:43.857907 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" podStartSLOduration=1.857890721 podStartE2EDuration="1.857890721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:43.855125747 +0000 UTC m=+1417.851082768" watchObservedRunningTime="2026-10-30 06:16:43.857890721 +0000 UTC m=+1417.853847730" Oct 30 06:16:44.069450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.062575 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb"] Oct 30 06:16:44.081332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.076988 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d630e74-h5pxb"] Oct 30 06:16:44.136364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.133942 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213d154b-abcb-4d71-9def-bc2b2aa6570f" path="/var/lib/kubelet/pods/213d154b-abcb-4d71-9def-bc2b2aa6570f/volumes" Oct 30 06:16:44.228272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.228236 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf"] Oct 30 06:16:44.235004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.234946 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.254533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.254411 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-584c9d-openshift-default-dockercfg-vjbsl\"" Oct 30 06:16:44.261022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.260990 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf"] Oct 30 06:16:44.344387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.342436 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-certs\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.344387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.342562 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-podinfo\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.344387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.342595 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da87567-9dee-47ec-9bfd-52f7cb332923-istiod-ca-cert\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.344387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.342656 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52lc\" (UniqueName: \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-kube-api-access-g52lc\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.344387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.342745 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-socket\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.344387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.342775 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-credential-socket\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.344387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.342813 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-data\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.344387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.342904 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-envoy\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.344387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.342995 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-token\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.444271 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-podinfo\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.444346 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da87567-9dee-47ec-9bfd-52f7cb332923-istiod-ca-cert\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.444414 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52lc\" (UniqueName: \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-kube-api-access-g52lc\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.444499 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-socket\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.444533 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-credential-socket\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.444570 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-data\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.444654 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-envoy\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.444757 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-token\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.444835 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-certs\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.445369 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-certs\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.446643 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-credential-socket\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.447419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.447267 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da87567-9dee-47ec-9bfd-52f7cb332923-istiod-ca-cert\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.448196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.447920 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-socket\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.458336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.452487 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-data\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.488335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.477667 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-envoy\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.508339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.500796 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-podinfo\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.508339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.505882 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-token\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.540329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.539250 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52lc\" (UniqueName: \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-kube-api-access-g52lc\") pod \"e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.592342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.585699 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:44.768733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.768651 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:44.778461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.775162 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:44.850075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.849983 5001 generic.go:356] "Generic (PLEG): container finished" podID="fdea9c7b-6f75-46ba-a1f6-9771c6358d4d" containerID="f9f092ce75f21928c6ddb7b0a3c24b325249e6911f891ef23d00c29462c6d081" exitCode=0 Oct 30 06:16:44.851432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.851404 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" event={"ID":"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d","Type":"ContainerDied","Data":"f9f092ce75f21928c6ddb7b0a3c24b325249e6911f891ef23d00c29462c6d081"} Oct 30 06:16:44.851575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.851503 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:44.859368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:44.853783 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:16:45.184721 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:16:45.184689 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da87567_9dee_47ec_9bfd_52f7cb332923.slice/crio-a666ecac18f820d8b593271b43608fa58521c0d87ca1d57bab6a8a1d820689d3 WatchSource:0}: Error finding container a666ecac18f820d8b593271b43608fa58521c0d87ca1d57bab6a8a1d820689d3: Status 404 returned error can't find the container with id a666ecac18f820d8b593271b43608fa58521c0d87ca1d57bab6a8a1d820689d3 Oct 30 06:16:45.185635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:45.185608 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf"] Oct 30 06:16:45.187359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:45.187329 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:45.187413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:45.187391 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:45.187499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:45.187419 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:45.861059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:45.861020 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" event={"ID":"5da87567-9dee-47ec-9bfd-52f7cb332923","Type":"ContainerStarted","Data":"2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d"} Oct 30 06:16:45.861289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:45.861089 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" event={"ID":"5da87567-9dee-47ec-9bfd-52f7cb332923","Type":"ContainerStarted","Data":"a666ecac18f820d8b593271b43608fa58521c0d87ca1d57bab6a8a1d820689d3"} Oct 30 06:16:45.908916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:45.908858 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" podStartSLOduration=1.908839986 podStartE2EDuration="1.908839986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:45.907649035 +0000 UTC m=+1419.903606058" watchObservedRunningTime="2026-10-30 06:16:45.908839986 +0000 UTC m=+1419.904796997" Oct 30 06:16:46.105949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.105580 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:46.232894 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.232801 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-xks-ca\") pod \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " Oct 30 06:16:46.233452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.232941 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-kube-api-access-dwwks\" (UniqueName: \"kubernetes.io/projected/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-kube-api-access-dwwks\") pod \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " Oct 30 06:16:46.233452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.233030 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-openshift-ca\") pod \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\" (UID: \"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d\") " Oct 30 06:16:46.233570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.233451 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-openshift-ca" pod "fdea9c7b-6f75-46ba-a1f6-9771c6358d4d" (UID: "fdea9c7b-6f75-46ba-a1f6-9771c6358d4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:46.233767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.233749 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:46.235349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.235291 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-kube-api-access-dwwks" pod "fdea9c7b-6f75-46ba-a1f6-9771c6358d4d" (UID: "fdea9c7b-6f75-46ba-a1f6-9771c6358d4d"). InnerVolumeSpecName "kube-api-access-dwwks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:46.235441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.235299 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-xks-ca" pod "fdea9c7b-6f75-46ba-a1f6-9771c6358d4d" (UID: "fdea9c7b-6f75-46ba-a1f6-9771c6358d4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:46.334974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.334936 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwwks\" (UniqueName: \"kubernetes.io/projected/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-kube-api-access-dwwks\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:46.334974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.334970 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:46.586230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.586149 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:46.591992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.591961 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:46.871175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.871144 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" Oct 30 06:16:46.871175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.871154 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p" event={"ID":"fdea9c7b-6f75-46ba-a1f6-9771c6358d4d","Type":"ContainerDied","Data":"30d28aa3dd31c589181a67221cf712b2bdd3c0d004085b26e6503f4dea9c9c95"} Oct 30 06:16:46.871497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.871215 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30d28aa3dd31c589181a67221cf712b2bdd3c0d004085b26e6503f4dea9c9c95" Oct 30 06:16:46.871796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.871685 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:46.872957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:46.872936 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:16:47.377409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.377371 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:16:47.465975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.465869 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-token\" (UniqueName: \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-token\") pod \"24b399b8-2225-4498-aa8f-e5ea451f7131\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " Oct 30 06:16:47.465975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.465942 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-certs\") pod \"24b399b8-2225-4498-aa8f-e5ea451f7131\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " Oct 30 06:16:47.466211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.466019 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-podinfo\") pod \"24b399b8-2225-4498-aa8f-e5ea451f7131\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " Oct 30 06:16:47.466211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.466135 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b399b8-2225-4498-aa8f-e5ea451f7131-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b399b8-2225-4498-aa8f-e5ea451f7131-istiod-ca-cert\") pod \"24b399b8-2225-4498-aa8f-e5ea451f7131\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " Oct 30 06:16:47.466211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.466193 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-kube-api-access-crxhc\" (UniqueName: \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-kube-api-access-crxhc\") pod \"24b399b8-2225-4498-aa8f-e5ea451f7131\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " Oct 30 06:16:47.466426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.466328 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-envoy\") pod \"24b399b8-2225-4498-aa8f-e5ea451f7131\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " Oct 30 06:16:47.466515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.466496 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-credential-socket\") pod \"24b399b8-2225-4498-aa8f-e5ea451f7131\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " Oct 30 06:16:47.466578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.466537 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-data\") pod \"24b399b8-2225-4498-aa8f-e5ea451f7131\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " Oct 30 06:16:47.466629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.466593 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-socket\") pod \"24b399b8-2225-4498-aa8f-e5ea451f7131\" (UID: \"24b399b8-2225-4498-aa8f-e5ea451f7131\") " Oct 30 06:16:47.467722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.467368 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-socket" pod "24b399b8-2225-4498-aa8f-e5ea451f7131" (UID: "24b399b8-2225-4498-aa8f-e5ea451f7131"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:47.467722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.467403 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b399b8-2225-4498-aa8f-e5ea451f7131-istiod-ca-cert" pod "24b399b8-2225-4498-aa8f-e5ea451f7131" (UID: "24b399b8-2225-4498-aa8f-e5ea451f7131"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:47.467722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.467571 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-credential-socket" pod "24b399b8-2225-4498-aa8f-e5ea451f7131" (UID: "24b399b8-2225-4498-aa8f-e5ea451f7131"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:47.467722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.467581 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-data" pod "24b399b8-2225-4498-aa8f-e5ea451f7131" (UID: "24b399b8-2225-4498-aa8f-e5ea451f7131"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:47.468029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.467733 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-certs" pod "24b399b8-2225-4498-aa8f-e5ea451f7131" (UID: "24b399b8-2225-4498-aa8f-e5ea451f7131"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:47.470684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.470659 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-envoy" pod "24b399b8-2225-4498-aa8f-e5ea451f7131" (UID: "24b399b8-2225-4498-aa8f-e5ea451f7131"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:47.470792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.470757 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-podinfo" pod "24b399b8-2225-4498-aa8f-e5ea451f7131" (UID: "24b399b8-2225-4498-aa8f-e5ea451f7131"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:16:47.472452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.472429 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-kube-api-access-crxhc" pod "24b399b8-2225-4498-aa8f-e5ea451f7131" (UID: "24b399b8-2225-4498-aa8f-e5ea451f7131"). InnerVolumeSpecName "kube-api-access-crxhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:47.472617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.472509 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-token" pod "24b399b8-2225-4498-aa8f-e5ea451f7131" (UID: "24b399b8-2225-4498-aa8f-e5ea451f7131"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:47.593337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.593143 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:47.593337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.593178 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:47.593337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.593193 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:47.593337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.593207 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:47.593337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.593219 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:47.593337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.593231 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b399b8-2225-4498-aa8f-e5ea451f7131-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:47.593337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.593243 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b399b8-2225-4498-aa8f-e5ea451f7131-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:47.593337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.593256 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b399b8-2225-4498-aa8f-e5ea451f7131-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:47.593337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.593269 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crxhc\" (UniqueName: \"kubernetes.io/projected/24b399b8-2225-4498-aa8f-e5ea451f7131-kube-api-access-crxhc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:47.881401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.881363 5001 generic.go:356] "Generic (PLEG): container finished" podID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerID="b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d" exitCode=0 Oct 30 06:16:47.881580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.881424 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" Oct 30 06:16:47.881580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.881447 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" event={"ID":"24b399b8-2225-4498-aa8f-e5ea451f7131","Type":"ContainerDied","Data":"b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d"} Oct 30 06:16:47.881580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.881513 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn" event={"ID":"24b399b8-2225-4498-aa8f-e5ea451f7131","Type":"ContainerDied","Data":"96a4b14a7d9d97a510050f694037f281d0632ae8e9364828ae12a01f01844f87"} Oct 30 06:16:47.881771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.881591 5001 scope.go:122] "RemoveContainer" containerID="b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d" Oct 30 06:16:47.895152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.895131 5001 scope.go:122] "RemoveContainer" containerID="e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0" Oct 30 06:16:47.916257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.916237 5001 scope.go:122] "RemoveContainer" containerID="b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d" Oct 30 06:16:47.916957 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:16:47.916921 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d\": container with ID starting with b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d not found: ID does not exist" containerID="b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d" Oct 30 06:16:47.917058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.916967 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d"} err="failed to get container status \"b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d\": rpc error: code = NotFound desc = could not find container \"b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d\": container with ID starting with b78af1d0d3d2b1dc2c845babef466261030d4a293d85be510d437c21ffb7205d not found: ID does not exist" Oct 30 06:16:47.917058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.916991 5001 scope.go:122] "RemoveContainer" containerID="e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0" Oct 30 06:16:47.917897 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:16:47.917872 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0\": container with ID starting with e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0 not found: ID does not exist" containerID="e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0" Oct 30 06:16:47.917993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.917904 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0"} err="failed to get container status \"e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0\": rpc error: code = NotFound desc = could not find container \"e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0\": container with ID starting with e5e5cd25e9c38d957ffe8f5bbc3dc39bed7d0425e5f192e431df279881be70b0 not found: ID does not exist" Oct 30 06:16:47.941363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.941332 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br"] Oct 30 06:16:47.941589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.941567 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" containerID="cri-o://4c7e72eeef02b3a136ec36ccbb5d2dd2c98e7aa4f836f56d6080899d86199529" gracePeriod=30 Oct 30 06:16:47.966350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.966296 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn"] Oct 30 06:16:47.973631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:47.973352 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c15d3a-openshift-default-695c69b767-fq8qn"] Oct 30 06:16:48.122649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.122595 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" path="/var/lib/kubelet/pods/24b399b8-2225-4498-aa8f-e5ea451f7131/volumes" Oct 30 06:16:48.801228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.792843 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7"] Oct 30 06:16:48.801228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.793916 5001 cpu_manager.go:414] "containerMap: removing container" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" Oct 30 06:16:48.801228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.793940 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" Oct 30 06:16:48.801228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.793992 5001 cpu_manager.go:414] "containerMap: removing container" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" Oct 30 06:16:48.801228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.794001 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" Oct 30 06:16:48.801228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.794018 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fdea9c7b-6f75-46ba-a1f6-9771c6358d4d" containerName="revoke-keys" Oct 30 06:16:48.801228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.794025 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdea9c7b-6f75-46ba-a1f6-9771c6358d4d" containerName="revoke-keys" Oct 30 06:16:48.801228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.794212 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" Oct 30 06:16:48.801228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.794231 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdea9c7b-6f75-46ba-a1f6-9771c6358d4d" containerName="revoke-keys" Oct 30 06:16:48.974342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.968163 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7"] Oct 30 06:16:48.974342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.968345 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:16:48.985461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:48.985429 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-33c99c-dockercfg-plrl5\"" Oct 30 06:16:49.042288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.042249 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-33c99c\"" Oct 30 06:16:49.058383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.058272 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6"] Oct 30 06:16:49.069738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.069708 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da240f3f-bpbj6"] Oct 30 06:16:49.069914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.069840 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62zb\" (UniqueName: \"kubernetes.io/projected/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-kube-api-access-s62zb\") pod \"payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7\" (UID: \"ed134af0-faeb-48d7-a07d-47e57c8aa3d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:16:49.070080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.070057 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7\" (UID: \"ed134af0-faeb-48d7-a07d-47e57c8aa3d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:16:49.171556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.171526 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7\" (UID: \"ed134af0-faeb-48d7-a07d-47e57c8aa3d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:16:49.171734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.171700 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62zb\" (UniqueName: \"kubernetes.io/projected/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-kube-api-access-s62zb\") pod \"payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7\" (UID: \"ed134af0-faeb-48d7-a07d-47e57c8aa3d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:16:49.172366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.172338 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7\" (UID: \"ed134af0-faeb-48d7-a07d-47e57c8aa3d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:16:49.203599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.203572 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62zb\" (UniqueName: \"kubernetes.io/projected/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-kube-api-access-s62zb\") pod \"payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7\" (UID: \"ed134af0-faeb-48d7-a07d-47e57c8aa3d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:16:49.300402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.300363 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:16:49.335025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.334994 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw"] Oct 30 06:16:49.336232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.336206 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b399b8-2225-4498-aa8f-e5ea451f7131" containerName="istio-proxy" Oct 30 06:16:49.341246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.341224 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.369949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.369850 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw"] Oct 30 06:16:49.375047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.374190 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21141de6-8bbc-4245-9c58-d4218a32169c-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.375047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.374253 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68r4b\" (UniqueName: \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-kube-api-access-68r4b\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.375047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.374327 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.375047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.374382 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.375047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.374419 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21141de6-8bbc-4245-9c58-d4218a32169c-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.375047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.374512 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.375047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.374549 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.375047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.374568 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.375047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.374601 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.438178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.438140 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0fab6d-openshift-default-dockercfg-7pz45\"" Oct 30 06:16:49.517327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.517075 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.517696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.517668 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.518042 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.518094 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.518134 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.518201 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.518413 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21141de6-8bbc-4245-9c58-d4218a32169c-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.518481 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68r4b\" (UniqueName: \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-kube-api-access-68r4b\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.518541 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.518651 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.518726 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21141de6-8bbc-4245-9c58-d4218a32169c-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.519880 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21141de6-8bbc-4245-9c58-d4218a32169c-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.520590 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.533333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.521135 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.593791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.571661 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21141de6-8bbc-4245-9c58-d4218a32169c-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.593791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.588650 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.667784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.667754 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.671842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.670360 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68r4b\" (UniqueName: \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-kube-api-access-68r4b\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:49.685023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.682230 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7"] Oct 30 06:16:49.966943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:49.966907 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:50.071014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:50.070975 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" event={"ID":"ed134af0-faeb-48d7-a07d-47e57c8aa3d0","Type":"ContainerStarted","Data":"d72d9d5e2612170a03ae6fc8e5af5705d6e2865b3d7e9e63e1e9c8493383d2cd"} Oct 30 06:16:50.122294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:50.122265 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf4e00d-0125-46b9-b0d3-1a98ca43d1c5" path="/var/lib/kubelet/pods/abf4e00d-0125-46b9-b0d3-1a98ca43d1c5/volumes" Oct 30 06:16:50.400072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:50.398988 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw"] Oct 30 06:16:50.400072 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:16:50.399106 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21141de6_8bbc_4245_9c58_d4218a32169c.slice/crio-5d1b36a83565190e2224e6aad01ff75b547b6b2bbda7760b309cec064360de3f WatchSource:0}: Error finding container 5d1b36a83565190e2224e6aad01ff75b547b6b2bbda7760b309cec064360de3f: Status 404 returned error can't find the container with id 5d1b36a83565190e2224e6aad01ff75b547b6b2bbda7760b309cec064360de3f Oct 30 06:16:50.402704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:50.402662 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:50.402839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:50.402751 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:50.402839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:50.402794 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:51.081851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:51.081813 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" event={"ID":"21141de6-8bbc-4245-9c58-d4218a32169c","Type":"ContainerStarted","Data":"4523f586adf5e1f1d1d98c5cb05be0d3bda229178280f153e194ae473e2b2959"} Oct 30 06:16:51.082403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:51.081882 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" event={"ID":"21141de6-8bbc-4245-9c58-d4218a32169c","Type":"ContainerStarted","Data":"5d1b36a83565190e2224e6aad01ff75b547b6b2bbda7760b309cec064360de3f"} Oct 30 06:16:51.084200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:51.084170 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" event={"ID":"ed134af0-faeb-48d7-a07d-47e57c8aa3d0","Type":"ContainerStarted","Data":"bc9b368fc57e7cfcae52c452c974925aaf565a4b45b1a4cf5d0bb5a1ce47e57d"} Oct 30 06:16:51.084391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:51.084268 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:16:51.131101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:51.131040 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" podStartSLOduration=2.131023875 podStartE2EDuration="2.131023875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:51.128172932 +0000 UTC m=+1425.124129943" watchObservedRunningTime="2026-10-30 06:16:51.131023875 +0000 UTC m=+1425.126980953" Oct 30 06:16:51.175733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:51.175682 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" podStartSLOduration=3.175668306 podStartE2EDuration="3.175668306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:51.173298117 +0000 UTC m=+1425.169255127" watchObservedRunningTime="2026-10-30 06:16:51.175668306 +0000 UTC m=+1425.171625315" Oct 30 06:16:51.967730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:51.967697 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:51.973610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:51.973579 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:52.092838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:52.092797 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:52.093894 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:52.093871 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:16:53.105902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.105861 5001 generic.go:356] "Generic (PLEG): container finished" podID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerID="4c7e72eeef02b3a136ec36ccbb5d2dd2c98e7aa4f836f56d6080899d86199529" exitCode=0 Oct 30 06:16:53.106409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.105938 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" event={"ID":"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39","Type":"ContainerDied","Data":"4c7e72eeef02b3a136ec36ccbb5d2dd2c98e7aa4f836f56d6080899d86199529"} Oct 30 06:16:53.106409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.105998 5001 scope.go:122] "RemoveContainer" containerID="042d9c4c3e3598227335bc182d22baa9e8e698eef9278c5f3f985434feff7711" Oct 30 06:16:53.246461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.246432 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:16:53.265068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.265036 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6"] Oct 30 06:16:53.341657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.341617 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-token\" (UniqueName: \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-token\") pod \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " Oct 30 06:16:53.341856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.341691 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-envoy\") pod \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " Oct 30 06:16:53.341856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.341783 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-data\") pod \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " Oct 30 06:16:53.341856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.341819 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-kube-api-access-j796k\" (UniqueName: \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-kube-api-access-j796k\") pod \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " Oct 30 06:16:53.342084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.341911 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-podinfo\") pod \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " Oct 30 06:16:53.342084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.341973 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-socket\") pod \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " Oct 30 06:16:53.342084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.342004 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-certs\") pod \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " Oct 30 06:16:53.342266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.342099 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istiod-ca-cert\") pod \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " Oct 30 06:16:53.342266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.342192 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-credential-socket\") pod \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\" (UID: \"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39\") " Oct 30 06:16:53.343349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.342905 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-socket" pod "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" (UID: "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:53.343349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.343043 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-certs" pod "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" (UID: "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:53.343606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.343573 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istiod-ca-cert" pod "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" (UID: "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:53.343758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.343736 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-credential-socket" pod "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" (UID: "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:53.344836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.344711 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-data" pod "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" (UID: "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:53.347653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.347626 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-token" pod "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" (UID: "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:53.349094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.349057 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-envoy" pod "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" (UID: "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:53.349528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.349496 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-podinfo" pod "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" (UID: "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:16:53.354490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.353822 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-kube-api-access-j796k" pod "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" (UID: "d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39"). InnerVolumeSpecName "kube-api-access-j796k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:53.445089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.445050 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:53.445089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.445087 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j796k\" (UniqueName: \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-kube-api-access-j796k\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:53.445326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.445105 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:53.445326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.445119 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:53.445326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.445128 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:53.445326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.445137 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:53.445326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.445146 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:53.445326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.445154 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:53.445326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:53.445162 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:54.203431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:54.197988 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" Oct 30 06:16:54.203431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:54.201410 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br" event={"ID":"d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39","Type":"ContainerDied","Data":"118fc22151ac3a3548e13ce23ffc8a0205c4f4225bc73130cad0009b14cd0430"} Oct 30 06:16:54.203431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:54.201494 5001 scope.go:122] "RemoveContainer" containerID="4c7e72eeef02b3a136ec36ccbb5d2dd2c98e7aa4f836f56d6080899d86199529" Oct 30 06:16:54.273594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:54.273465 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br"] Oct 30 06:16:54.285368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:54.285335 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ba44b248-openshift-default-8597d4c685-2m4br"] Oct 30 06:16:55.622415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.622382 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd"] Oct 30 06:16:55.623057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.623034 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" Oct 30 06:16:55.623057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.623053 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" Oct 30 06:16:55.623256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.623082 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" Oct 30 06:16:55.623256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.623087 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" Oct 30 06:16:55.623256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.623244 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" Oct 30 06:16:55.623401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.623266 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" containerName="istio-proxy" Oct 30 06:16:55.626816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.626799 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:16:55.642395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.642371 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0fab6d\"" Oct 30 06:16:55.656931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.656898 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0fab6d-dockercfg-wqt5d\"" Oct 30 06:16:55.675223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.675182 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd"] Oct 30 06:16:55.722342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.719572 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd\" (UID: \"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:16:55.722342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.719775 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pv6h\" (UniqueName: \"kubernetes.io/projected/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-kube-api-access-5pv6h\") pod \"payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd\" (UID: \"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:16:55.821329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.806193 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44"] Oct 30 06:16:55.821329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.815886 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:16:55.867389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.866573 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pv6h\" (UniqueName: \"kubernetes.io/projected/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-kube-api-access-5pv6h\") pod \"payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd\" (UID: \"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:16:55.867389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.866887 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd\" (UID: \"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:16:55.867921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.867883 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd\" (UID: \"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:16:55.928784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.926212 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44"] Oct 30 06:16:55.977337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.969445 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkttk\" (UniqueName: \"kubernetes.io/projected/58b8efdd-c28a-4818-a8a5-fbd3a422669e-kube-api-access-bkttk\") pod \"payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44\" (UID: \"58b8efdd-c28a-4818-a8a5-fbd3a422669e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:16:55.977337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:55.969580 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b8efdd-c28a-4818-a8a5-fbd3a422669e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44\" (UID: \"58b8efdd-c28a-4818-a8a5-fbd3a422669e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:16:56.053374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:56.053112 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pv6h\" (UniqueName: \"kubernetes.io/projected/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-kube-api-access-5pv6h\") pod \"payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd\" (UID: \"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:16:56.082378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:56.075419 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkttk\" (UniqueName: \"kubernetes.io/projected/58b8efdd-c28a-4818-a8a5-fbd3a422669e-kube-api-access-bkttk\") pod \"payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44\" (UID: \"58b8efdd-c28a-4818-a8a5-fbd3a422669e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:16:56.082378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:56.075523 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b8efdd-c28a-4818-a8a5-fbd3a422669e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44\" (UID: \"58b8efdd-c28a-4818-a8a5-fbd3a422669e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:16:56.086345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:56.083710 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b8efdd-c28a-4818-a8a5-fbd3a422669e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44\" (UID: \"58b8efdd-c28a-4818-a8a5-fbd3a422669e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:16:56.108903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:56.108861 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkttk\" (UniqueName: \"kubernetes.io/projected/58b8efdd-c28a-4818-a8a5-fbd3a422669e-kube-api-access-bkttk\") pod \"payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44\" (UID: \"58b8efdd-c28a-4818-a8a5-fbd3a422669e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:16:56.118265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:56.118232 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39" path="/var/lib/kubelet/pods/d6ac2989-2bfd-4b85-9fbf-c3c04f3b2c39/volumes" Oct 30 06:16:56.241660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:56.241555 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:16:56.315109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:56.315077 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:16:56.592394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:56.592300 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd"] Oct 30 06:16:56.710413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:56.710386 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44"] Oct 30 06:16:57.227817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:57.227777 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" event={"ID":"58b8efdd-c28a-4818-a8a5-fbd3a422669e","Type":"ContainerStarted","Data":"8e73d25e7abc17d49bca2821642d6170c7bcdc82753f31da260fd04041ca8068"} Oct 30 06:16:57.228027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:57.227842 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" event={"ID":"58b8efdd-c28a-4818-a8a5-fbd3a422669e","Type":"ContainerStarted","Data":"d4169824df00c0019b1badeb97c2d35a1cd6673490b38475321ed526f1530563"} Oct 30 06:16:57.228027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:57.227868 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:16:57.229724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:57.229699 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" event={"ID":"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a","Type":"ContainerStarted","Data":"2f80ca4b9d1c86cfa3fd5ea912aa3af273f31c13e2d2e1d3b98c097d4ed9b825"} Oct 30 06:16:57.229851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:57.229744 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" event={"ID":"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a","Type":"ContainerStarted","Data":"e75cc44865e097563633585a4058dc8edc2d41fd460bf8f38a2c7b4cf194c3d5"} Oct 30 06:16:57.229851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:57.229770 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:16:57.272991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:57.272933 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" podStartSLOduration=2.272918347 podStartE2EDuration="2.272918347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:57.269537094 +0000 UTC m=+1431.265494118" watchObservedRunningTime="2026-10-30 06:16:57.272918347 +0000 UTC m=+1431.268875364" Oct 30 06:16:57.312069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:57.312016 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" podStartSLOduration=2.312001563 podStartE2EDuration="2.312001563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:57.310935789 +0000 UTC m=+1431.306892799" watchObservedRunningTime="2026-10-30 06:16:57.312001563 +0000 UTC m=+1431.307958573" Oct 30 06:16:58.265806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:58.265759 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" podUID="2138fd05-a058-4a27-8b6f-4c43b1f5de9c" containerName="payload-pre-processing" containerID="cri-o://0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093" gracePeriod=25 Oct 30 06:16:58.643345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:58.643296 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:16:58.728227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:58.728187 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-kube-api-access-qqb46\" (UniqueName: \"kubernetes.io/projected/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-kube-api-access-qqb46\") pod \"2138fd05-a058-4a27-8b6f-4c43b1f5de9c\" (UID: \"2138fd05-a058-4a27-8b6f-4c43b1f5de9c\") " Oct 30 06:16:58.728507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:58.728483 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-plugins-config-volume\") pod \"2138fd05-a058-4a27-8b6f-4c43b1f5de9c\" (UID: \"2138fd05-a058-4a27-8b6f-4c43b1f5de9c\") " Oct 30 06:16:58.728921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:58.728893 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-plugins-config-volume" pod "2138fd05-a058-4a27-8b6f-4c43b1f5de9c" (UID: "2138fd05-a058-4a27-8b6f-4c43b1f5de9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:58.729349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:58.729326 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:58.730726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:58.730699 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-kube-api-access-qqb46" pod "2138fd05-a058-4a27-8b6f-4c43b1f5de9c" (UID: "2138fd05-a058-4a27-8b6f-4c43b1f5de9c"). InnerVolumeSpecName "kube-api-access-qqb46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:58.830275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:58.830189 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqb46\" (UniqueName: \"kubernetes.io/projected/2138fd05-a058-4a27-8b6f-4c43b1f5de9c-kube-api-access-qqb46\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:16:59.247394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:59.247359 5001 generic.go:356] "Generic (PLEG): container finished" podID="2138fd05-a058-4a27-8b6f-4c43b1f5de9c" containerID="0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093" exitCode=0 Oct 30 06:16:59.247544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:59.247410 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" Oct 30 06:16:59.247544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:59.247439 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" event={"ID":"2138fd05-a058-4a27-8b6f-4c43b1f5de9c","Type":"ContainerDied","Data":"0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093"} Oct 30 06:16:59.247544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:59.247498 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6" event={"ID":"2138fd05-a058-4a27-8b6f-4c43b1f5de9c","Type":"ContainerDied","Data":"02d688bdc8f9f0ad8562016a4576de3ddae697c0769afd5aec7f7d25aa66f40b"} Oct 30 06:16:59.247544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:59.247523 5001 scope.go:122] "RemoveContainer" containerID="0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093" Oct 30 06:16:59.260895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:59.260865 5001 scope.go:122] "RemoveContainer" containerID="0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093" Oct 30 06:16:59.261242 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:16:59.261221 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093\": container with ID starting with 0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093 not found: ID does not exist" containerID="0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093" Oct 30 06:16:59.261318 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:59.261252 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093"} err="failed to get container status \"0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093\": rpc error: code = NotFound desc = could not find container \"0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093\": container with ID starting with 0086a9a04cbb4d1296286674c9096ee0209aa8389a8b718a76c7ac0e7051e093 not found: ID does not exist" Oct 30 06:16:59.305009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:59.304969 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6"] Oct 30 06:16:59.317125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:16:59.317089 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7a606-6ff7ccdd55-4scp6"] Oct 30 06:17:00.120104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:00.120064 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2138fd05-a058-4a27-8b6f-4c43b1f5de9c" path="/var/lib/kubelet/pods/2138fd05-a058-4a27-8b6f-4c43b1f5de9c/volumes" Oct 30 06:17:02.141725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:02.141677 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:17:07.230688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:07.230650 5001 scope.go:122] "RemoveContainer" containerID="d009592cb5659b30e4bd7ac2cbc48888a8d884cd91037c939328ff2b9649c306" Oct 30 06:17:07.241509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:07.241488 5001 scope.go:122] "RemoveContainer" containerID="dc0682a31533419b47aea92100c3fd5a9aedb7d8cc900258c237207a56821b9d" Oct 30 06:17:07.252108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:07.252081 5001 scope.go:122] "RemoveContainer" containerID="95d1c71ded133ce53dd9ae12fbf589dfe29b914f645b49b2ccd3d2f0a6020673" Oct 30 06:17:07.262952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:07.262930 5001 scope.go:122] "RemoveContainer" containerID="3686f016abf86b8ae5ae489f4b0eeb86a1f40ce42666711be66c35b0ff193bde" Oct 30 06:17:08.240807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:08.240777 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:17:08.241289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:08.240860 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:17:08.267298 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:17:08.267258 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:17:09.207566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.207532 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m"] Oct 30 06:17:09.209180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.208708 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2138fd05-a058-4a27-8b6f-4c43b1f5de9c" containerName="payload-pre-processing" Oct 30 06:17:09.209180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.208735 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2138fd05-a058-4a27-8b6f-4c43b1f5de9c" containerName="payload-pre-processing" Oct 30 06:17:09.209180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.208921 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2138fd05-a058-4a27-8b6f-4c43b1f5de9c" containerName="payload-pre-processing" Oct 30 06:17:09.220244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.220034 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.235664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.235630 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m"] Oct 30 06:17:09.241087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.241060 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f50f6e-openshift-default-dockercfg-5m7g9\"" Oct 30 06:17:09.279882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.279845 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-credential-socket\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.279882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.279884 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-certs\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.280142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.280056 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9m74\" (UniqueName: \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-kube-api-access-m9m74\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.280142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.280120 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-socket\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.280262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.280172 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-token\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.280262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.280224 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-envoy\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.280393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.280298 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-podinfo\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.280393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.280352 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istiod-ca-cert\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.280544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.280438 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-data\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.422476 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-data\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.422600 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-credential-socket\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.422638 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-certs\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.422706 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9m74\" (UniqueName: \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-kube-api-access-m9m74\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.422752 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-socket\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.422803 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-token\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.422833 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-envoy\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.422941 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-podinfo\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.422985 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istiod-ca-cert\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.423930 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istiod-ca-cert\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.424216 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-data\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.424470 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-credential-socket\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.431331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.424696 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-certs\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.518398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.425297 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-socket\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.597334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.539085 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9m74\" (UniqueName: \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-kube-api-access-m9m74\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.597334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.596860 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-token\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.653915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.653730 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-envoy\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.661348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.655803 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-podinfo\") pod \"e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:09.850383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:09.848583 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:10.324505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:10.324480 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m"] Oct 30 06:17:10.326350 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:17:10.326325 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf072d364_6b3a_4b8b_9ae8_b9ff16503cda.slice/crio-24f06b7317d1369451bf823bdaa810e64d71bee9db15b41d405b772eff888cb9 WatchSource:0}: Error finding container 24f06b7317d1369451bf823bdaa810e64d71bee9db15b41d405b772eff888cb9: Status 404 returned error can't find the container with id 24f06b7317d1369451bf823bdaa810e64d71bee9db15b41d405b772eff888cb9 Oct 30 06:17:10.328814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:10.328779 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:10.328906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:10.328857 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:10.328977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:10.328902 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:10.689711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:10.688351 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" event={"ID":"f072d364-6b3a-4b8b-9ae8-b9ff16503cda","Type":"ContainerStarted","Data":"8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa"} Oct 30 06:17:10.689711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:10.688423 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" event={"ID":"f072d364-6b3a-4b8b-9ae8-b9ff16503cda","Type":"ContainerStarted","Data":"24f06b7317d1369451bf823bdaa810e64d71bee9db15b41d405b772eff888cb9"} Oct 30 06:17:10.774461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:10.768960 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" podStartSLOduration=1.7689339149999999 podStartE2EDuration="1.768933915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:10.75643749 +0000 UTC m=+1444.752394500" watchObservedRunningTime="2026-10-30 06:17:10.768933915 +0000 UTC m=+1444.764890926" Oct 30 06:17:10.849605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:10.849574 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:11.908331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:11.907751 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:12.709290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:12.709241 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:12.710293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:12.710268 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:17:15.343605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.343559 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j"] Oct 30 06:17:15.380618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.380584 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:17:15.393025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.392985 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j"] Oct 30 06:17:15.401793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.401743 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f50f6e\"" Oct 30 06:17:15.414676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.414641 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f50f6e-dockercfg-t79tl\"" Oct 30 06:17:15.465329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.454230 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a558ec8-ad54-423a-a67b-75b7dc613bdc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j\" (UID: \"9a558ec8-ad54-423a-a67b-75b7dc613bdc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:17:15.465873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.465841 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbcm9\" (UniqueName: \"kubernetes.io/projected/9a558ec8-ad54-423a-a67b-75b7dc613bdc-kube-api-access-nbcm9\") pod \"payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j\" (UID: \"9a558ec8-ad54-423a-a67b-75b7dc613bdc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:17:15.632149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.632115 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a558ec8-ad54-423a-a67b-75b7dc613bdc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j\" (UID: \"9a558ec8-ad54-423a-a67b-75b7dc613bdc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:17:15.632592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.632570 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbcm9\" (UniqueName: \"kubernetes.io/projected/9a558ec8-ad54-423a-a67b-75b7dc613bdc-kube-api-access-nbcm9\") pod \"payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j\" (UID: \"9a558ec8-ad54-423a-a67b-75b7dc613bdc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:17:15.633130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.633100 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a558ec8-ad54-423a-a67b-75b7dc613bdc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j\" (UID: \"9a558ec8-ad54-423a-a67b-75b7dc613bdc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:17:15.657617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.657589 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbcm9\" (UniqueName: \"kubernetes.io/projected/9a558ec8-ad54-423a-a67b-75b7dc613bdc-kube-api-access-nbcm9\") pod \"payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j\" (UID: \"9a558ec8-ad54-423a-a67b-75b7dc613bdc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:17:15.741139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:15.741099 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:17:16.067489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:16.067457 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j"] Oct 30 06:17:16.068528 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:17:16.068495 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a558ec8_ad54_423a_a67b_75b7dc613bdc.slice/crio-5c6c1df16e9ba07f106c4b1b29352a4d4d9bd78de76009e00349aeab58e48c05 WatchSource:0}: Error finding container 5c6c1df16e9ba07f106c4b1b29352a4d4d9bd78de76009e00349aeab58e48c05: Status 404 returned error can't find the container with id 5c6c1df16e9ba07f106c4b1b29352a4d4d9bd78de76009e00349aeab58e48c05 Oct 30 06:17:16.746235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:16.746198 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" event={"ID":"9a558ec8-ad54-423a-a67b-75b7dc613bdc","Type":"ContainerStarted","Data":"a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972"} Oct 30 06:17:16.746781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:16.746267 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" event={"ID":"9a558ec8-ad54-423a-a67b-75b7dc613bdc","Type":"ContainerStarted","Data":"5c6c1df16e9ba07f106c4b1b29352a4d4d9bd78de76009e00349aeab58e48c05"} Oct 30 06:17:16.746781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:16.746438 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:17:16.794416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:16.794358 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" podStartSLOduration=1.794338837 podStartE2EDuration="1.794338837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:16.789194658 +0000 UTC m=+1450.785151669" watchObservedRunningTime="2026-10-30 06:17:16.794338837 +0000 UTC m=+1450.790295848" Oct 30 06:17:21.669179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:21.669140 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:17:22.009429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:22.008427 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7"] Oct 30 06:17:24.049778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:24.049740 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq"] Oct 30 06:17:24.109335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:24.100632 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60978a8f-5gbzq"] Oct 30 06:17:24.124551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:24.124510 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2cb130b-ad14-4726-9a79-8317f4e9fbd3" path="/var/lib/kubelet/pods/b2cb130b-ad14-4726-9a79-8317f4e9fbd3/volumes" Oct 30 06:17:25.227177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:25.227125 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:17:25.426968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:25.426930 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 06:17:27.014344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:27.014278 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" podUID="ed134af0-faeb-48d7-a07d-47e57c8aa3d0" containerName="payload-processing" containerID="cri-o://bc9b368fc57e7cfcae52c452c974925aaf565a4b45b1a4cf5d0bb5a1ce47e57d" gracePeriod=25 Oct 30 06:17:27.786333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:27.782018 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:17:28.838998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:28.838908 5001 generic.go:356] "Generic (PLEG): container finished" podID="ed134af0-faeb-48d7-a07d-47e57c8aa3d0" containerID="bc9b368fc57e7cfcae52c452c974925aaf565a4b45b1a4cf5d0bb5a1ce47e57d" exitCode=0 Oct 30 06:17:28.848624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:28.839816 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" event={"ID":"ed134af0-faeb-48d7-a07d-47e57c8aa3d0","Type":"ContainerDied","Data":"bc9b368fc57e7cfcae52c452c974925aaf565a4b45b1a4cf5d0bb5a1ce47e57d"} Oct 30 06:17:29.584754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:29.584701 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:17:29.638330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:29.637495 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:17:29.767333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:29.754528 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-kube-api-access-s62zb\" (UniqueName: \"kubernetes.io/projected/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-kube-api-access-s62zb\") pod \"ed134af0-faeb-48d7-a07d-47e57c8aa3d0\" (UID: \"ed134af0-faeb-48d7-a07d-47e57c8aa3d0\") " Oct 30 06:17:29.767333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:29.754661 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-plugins-config-volume\") pod \"ed134af0-faeb-48d7-a07d-47e57c8aa3d0\" (UID: \"ed134af0-faeb-48d7-a07d-47e57c8aa3d0\") " Oct 30 06:17:29.794169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:29.794075 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-kube-api-access-s62zb" pod "ed134af0-faeb-48d7-a07d-47e57c8aa3d0" (UID: "ed134af0-faeb-48d7-a07d-47e57c8aa3d0"). InnerVolumeSpecName "kube-api-access-s62zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:29.815333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:29.812072 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-plugins-config-volume" pod "ed134af0-faeb-48d7-a07d-47e57c8aa3d0" (UID: "ed134af0-faeb-48d7-a07d-47e57c8aa3d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:29.862711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:29.862667 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62zb\" (UniqueName: \"kubernetes.io/projected/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-kube-api-access-s62zb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:29.881638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:29.881598 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed134af0-faeb-48d7-a07d-47e57c8aa3d0-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:29.941569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:29.941519 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" event={"ID":"ed134af0-faeb-48d7-a07d-47e57c8aa3d0","Type":"ContainerDied","Data":"d72d9d5e2612170a03ae6fc8e5af5705d6e2865b3d7e9e63e1e9c8493383d2cd"} Oct 30 06:17:29.941962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:29.941936 5001 scope.go:122] "RemoveContainer" containerID="bc9b368fc57e7cfcae52c452c974925aaf565a4b45b1a4cf5d0bb5a1ce47e57d" Oct 30 06:17:29.942879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:29.942862 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7" Oct 30 06:17:30.118990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:30.118497 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7"] Oct 30 06:17:30.139381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:30.139123 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-fbb7dd756-l8bj7"] Oct 30 06:17:32.224803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:32.224403 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed134af0-faeb-48d7-a07d-47e57c8aa3d0" path="/var/lib/kubelet/pods/ed134af0-faeb-48d7-a07d-47e57c8aa3d0/volumes" Oct 30 06:17:32.916473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:32.916434 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:17:32.916660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:32.916514 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:17:33.230803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:33.230623 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p"] Oct 30 06:17:33.246400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:33.245993 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7a606-nf55p"] Oct 30 06:17:34.127207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:34.127162 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdea9c7b-6f75-46ba-a1f6-9771c6358d4d" path="/var/lib/kubelet/pods/fdea9c7b-6f75-46ba-a1f6-9771c6358d4d/volumes" Oct 30 06:17:35.672629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.671709 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6"] Oct 30 06:17:35.672629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.671973 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" podUID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" containerName="istio-proxy" containerID="cri-o://61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a" gracePeriod=30 Oct 30 06:17:35.677900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.677873 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh"] Oct 30 06:17:35.678867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.678844 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ed134af0-faeb-48d7-a07d-47e57c8aa3d0" containerName="payload-processing" Oct 30 06:17:35.678867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.678868 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed134af0-faeb-48d7-a07d-47e57c8aa3d0" containerName="payload-processing" Oct 30 06:17:35.679060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.679013 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed134af0-faeb-48d7-a07d-47e57c8aa3d0" containerName="payload-processing" Oct 30 06:17:35.778615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.778545 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh"] Oct 30 06:17:35.778865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.778654 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:35.804186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.804149 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-31407e-openshift-default-dockercfg-htctq\"" Oct 30 06:17:35.905405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.904753 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-podinfo\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:35.905405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.904861 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-credential-socket\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:35.905405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.904905 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-token\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:35.905405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.904957 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-envoy\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:35.905405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.905008 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7cgw\" (UniqueName: \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-kube-api-access-q7cgw\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:35.905405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.905098 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-data\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:35.905405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.905193 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istiod-ca-cert\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:35.905405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.905229 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-socket\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:35.905405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:35.905276 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-certs\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.015478 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istiod-ca-cert\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.015553 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-socket\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.015601 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-certs\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.015695 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-podinfo\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.015780 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-credential-socket\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.015813 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-token\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.015846 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-envoy\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.015895 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7cgw\" (UniqueName: \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-kube-api-access-q7cgw\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.015988 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-data\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.016481 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-data\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.017166 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istiod-ca-cert\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.017439 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-socket\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.017667 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-certs\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.029388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.018962 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-credential-socket\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.106148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.106105 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7cgw\" (UniqueName: \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-kube-api-access-q7cgw\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.194804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.194380 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-envoy\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.196386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.196217 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-podinfo\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.274354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.274318 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-token\") pod \"e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.291004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.290707 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz"] Oct 30 06:17:36.291004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.290784 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2"] Oct 30 06:17:36.291688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.291394 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g"] Oct 30 06:17:36.292506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.291636 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" containerID="cri-o://32200783d1486cc072fe792cadf3ed115fc012ca07faa87b47b30c0c0569a3db" gracePeriod=30 Oct 30 06:17:36.292506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.292116 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerName="istio-proxy" containerID="cri-o://57f5e65e93e3fef5e23228f22ccb786a58bd46f2f5c3d58d71c5f26bfead1d93" gracePeriod=30 Oct 30 06:17:36.292506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.292373 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" podUID="63eca24a-c41d-4d53-b691-98d71d929310" containerName="istio-proxy" containerID="cri-o://2352bf41ac2e3e5cf43c05d395a8ffe99ce9a2d2611e487abf914879e3627df3" gracePeriod=30 Oct 30 06:17:36.298315 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.298233 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz"] Oct 30 06:17:36.298711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.298663 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerName="istio-proxy" containerID="cri-o://f97bcc009475f5b6960ac91e1838404d5943c34f480948f009226207e1050c60" gracePeriod=30 Oct 30 06:17:36.503951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.503902 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:36.979264 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:17:36.979213 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286fb2e3_76ff_44f6_aaf4_c537b0673ce5.slice/crio-defcea1e9c5975958bc9c0cd14b96bbaa3b404d897b5758814796a979043c3f1 WatchSource:0}: Error finding container defcea1e9c5975958bc9c0cd14b96bbaa3b404d897b5758814796a979043c3f1: Status 404 returned error can't find the container with id defcea1e9c5975958bc9c0cd14b96bbaa3b404d897b5758814796a979043c3f1 Oct 30 06:17:36.981854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.981829 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh"] Oct 30 06:17:36.983161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.983122 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:36.983271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.983212 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:36.983271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:36.983257 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:37.325044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:37.324998 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" event={"ID":"286fb2e3-76ff-44f6-aaf4-c537b0673ce5","Type":"ContainerStarted","Data":"18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30"} Oct 30 06:17:37.325236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:37.325072 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" event={"ID":"286fb2e3-76ff-44f6-aaf4-c537b0673ce5","Type":"ContainerStarted","Data":"defcea1e9c5975958bc9c0cd14b96bbaa3b404d897b5758814796a979043c3f1"} Oct 30 06:17:37.899371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:37.899336 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j"] Oct 30 06:17:37.899796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:37.899750 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerName="istio-proxy" containerID="cri-o://8664586e94dd95a6a6bec68f7173ef732a675683eff5112a4e4394f3089ed6fb" gracePeriod=30 Oct 30 06:17:38.134493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:38.134444 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:17:38.135014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:38.134507 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:17:38.338074 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:17:38.337898 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:17:38.414976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:38.414916 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" podStartSLOduration=3.414895562 podStartE2EDuration="3.414895562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:38.409250978 +0000 UTC m=+1472.405207989" watchObservedRunningTime="2026-10-30 06:17:38.414895562 +0000 UTC m=+1472.410852574" Oct 30 06:17:38.504288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:38.504255 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:38.511417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:38.511386 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:38.614481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:38.614432 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2"] Oct 30 06:17:38.618167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:38.614867 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" podUID="ad5ce104-9403-4712-8512-9845efc5ac51" containerName="istio-proxy" containerID="cri-o://7381bc7f6f15ac882500104dd3fb6dee705cc44f91cd5b3ce31950ef53a8b41b" gracePeriod=30 Oct 30 06:17:38.748770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:38.747180 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn"] Oct 30 06:17:38.748770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:38.747570 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" podUID="1bcadd15-1201-491a-8348-dd39b94f9bae" containerName="istio-proxy" containerID="cri-o://15ce3dc383cb1eb254b2aed7d792942023c02235cfb13dff6491c01b1abfeeac" gracePeriod=30 Oct 30 06:17:39.536279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:39.466607 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:39.536279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:39.486786 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:17:40.565340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:40.565095 5001 generic.go:356] "Generic (PLEG): container finished" podID="20c3319d-5388-4262-8abc-3bd921947b04" containerID="a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71" exitCode=0 Oct 30 06:17:40.568233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:40.566232 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" event={"ID":"20c3319d-5388-4262-8abc-3bd921947b04","Type":"ContainerDied","Data":"a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71"} Oct 30 06:17:40.570710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:40.570683 5001 scope.go:122] "RemoveContainer" containerID="a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71" Oct 30 06:17:40.573668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:40.573300 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:40.573668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:40.573426 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:40.573668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:40.573464 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:41.090761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.090730 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:17:41.186184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.186092 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-socket\") pod \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " Oct 30 06:17:41.186184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.186171 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-envoy\") pod \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " Oct 30 06:17:41.186478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.186200 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-token\" (UniqueName: \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-token\") pod \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " Oct 30 06:17:41.186478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.186255 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-credential-socket\") pod \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " Oct 30 06:17:41.186478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.186332 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-kube-api-access-ngtv9\" (UniqueName: \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-kube-api-access-ngtv9\") pod \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " Oct 30 06:17:41.186669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.186482 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-data\") pod \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " Oct 30 06:17:41.186669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.186533 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istiod-ca-cert\") pod \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " Oct 30 06:17:41.186669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.186569 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-certs\") pod \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " Oct 30 06:17:41.186669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.186607 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-podinfo\") pod \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\" (UID: \"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf\") " Oct 30 06:17:41.187702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.187649 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-credential-socket" pod "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" (UID: "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.187915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.187859 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-socket" pod "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" (UID: "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.189525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.189325 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-certs" pod "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" (UID: "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.189525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.189465 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istiod-ca-cert" pod "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" (UID: "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:41.189525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.189504 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-data" pod "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" (UID: "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.193241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.193201 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-kube-api-access-ngtv9" pod "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" (UID: "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf"). InnerVolumeSpecName "kube-api-access-ngtv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.193862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.193828 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-envoy" pod "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" (UID: "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.200824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.200786 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-podinfo" pod "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" (UID: "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:41.202514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.202472 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-token" pod "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" (UID: "2c3a428c-c55f-4ea6-b9e2-be1dce741eaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.295276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.295114 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.295276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.295157 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.295276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.295172 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.295276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.295185 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.295276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.295197 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.295276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.295209 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.295276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.295222 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.295276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.295235 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.295276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.295250 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngtv9\" (UniqueName: \"kubernetes.io/projected/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf-kube-api-access-ngtv9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.662661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.649231 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" event={"ID":"20c3319d-5388-4262-8abc-3bd921947b04","Type":"ContainerStarted","Data":"a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09"} Oct 30 06:17:41.666670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.666478 5001 generic.go:356] "Generic (PLEG): container finished" podID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerID="57f5e65e93e3fef5e23228f22ccb786a58bd46f2f5c3d58d71c5f26bfead1d93" exitCode=0 Oct 30 06:17:41.666670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.666590 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" event={"ID":"1a443ba6-2255-490b-90b2-4922b8eaf75d","Type":"ContainerDied","Data":"57f5e65e93e3fef5e23228f22ccb786a58bd46f2f5c3d58d71c5f26bfead1d93"} Oct 30 06:17:41.794428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.778613 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:17:41.894340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.797922 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:17:41.894340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.797988 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:17:41.894340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.806254 5001 generic.go:356] "Generic (PLEG): container finished" podID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerID="f97bcc009475f5b6960ac91e1838404d5943c34f480948f009226207e1050c60" exitCode=0 Oct 30 06:17:41.894340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.806392 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" event={"ID":"43cec6bc-bd19-4c3b-9520-8fd3eb41c005","Type":"ContainerDied","Data":"f97bcc009475f5b6960ac91e1838404d5943c34f480948f009226207e1050c60"} Oct 30 06:17:41.894340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.850574 5001 generic.go:356] "Generic (PLEG): container finished" podID="63eca24a-c41d-4d53-b691-98d71d929310" containerID="2352bf41ac2e3e5cf43c05d395a8ffe99ce9a2d2611e487abf914879e3627df3" exitCode=0 Oct 30 06:17:41.894340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.850686 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" event={"ID":"63eca24a-c41d-4d53-b691-98d71d929310","Type":"ContainerDied","Data":"2352bf41ac2e3e5cf43c05d395a8ffe99ce9a2d2611e487abf914879e3627df3"} Oct 30 06:17:41.911330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.902011 5001 generic.go:356] "Generic (PLEG): container finished" podID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" containerID="61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a" exitCode=0 Oct 30 06:17:41.911330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.902229 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" event={"ID":"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf","Type":"ContainerDied","Data":"61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a"} Oct 30 06:17:41.911330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.902288 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" event={"ID":"2c3a428c-c55f-4ea6-b9e2-be1dce741eaf","Type":"ContainerDied","Data":"a380fd09c6a37efbb94d276e642963c72dd77a146bf1aa43bf7fa2add8e8395c"} Oct 30 06:17:41.911330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.902338 5001 scope.go:122] "RemoveContainer" containerID="61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a" Oct 30 06:17:41.911330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.902536 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6" Oct 30 06:17:41.933611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.920835 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2"] Oct 30 06:17:41.933611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:41.921140 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" containerID="cri-o://6e62ce5bdcd3faefa71532b83323531c5d0d9251562db1a2df7cf641e4e919ae" gracePeriod=30 Oct 30 06:17:42.078055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.075537 5001 generic.go:356] "Generic (PLEG): container finished" podID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerID="32200783d1486cc072fe792cadf3ed115fc012ca07faa87b47b30c0c0569a3db" exitCode=0 Oct 30 06:17:42.078055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.075776 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" event={"ID":"9d1d0f14-43a9-4609-a7c7-de84d0c47758","Type":"ContainerDied","Data":"32200783d1486cc072fe792cadf3ed115fc012ca07faa87b47b30c0c0569a3db"} Oct 30 06:17:42.080419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.078670 5001 generic.go:356] "Generic (PLEG): container finished" podID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerID="2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d" exitCode=0 Oct 30 06:17:42.080419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.078951 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" event={"ID":"5da87567-9dee-47ec-9bfd-52f7cb332923","Type":"ContainerDied","Data":"2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d"} Oct 30 06:17:42.080419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.079548 5001 scope.go:122] "RemoveContainer" containerID="2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d" Oct 30 06:17:42.193451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.185557 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:42.193451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.185697 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:42.193451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.185761 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:42.324141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.324117 5001 scope.go:122] "RemoveContainer" containerID="61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a" Oct 30 06:17:42.327524 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:17:42.324803 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a\": container with ID starting with 61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a not found: ID does not exist" containerID="61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a" Oct 30 06:17:42.327524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.324841 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a"} err="failed to get container status \"61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a\": rpc error: code = NotFound desc = could not find container \"61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a\": container with ID starting with 61358c9d4544ac04c1e8867e7699236b066fcf8c25c76a94d0b1849ee9a8042a not found: ID does not exist" Oct 30 06:17:42.327524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.324865 5001 scope.go:122] "RemoveContainer" containerID="df8c3f90fe3de3afc2bd82234e12a9a7bd1c48b0185a54e845b3940f62dfb7f4" Oct 30 06:17:42.413026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.413007 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:17:42.439286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.439241 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6"] Oct 30 06:17:42.446753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.446727 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-hhqc6"] Oct 30 06:17:42.584342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.583162 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-credential-socket\") pod \"1a443ba6-2255-490b-90b2-4922b8eaf75d\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " Oct 30 06:17:42.584342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.583231 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-certs\") pod \"1a443ba6-2255-490b-90b2-4922b8eaf75d\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " Oct 30 06:17:42.584342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.583323 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-envoy\") pod \"1a443ba6-2255-490b-90b2-4922b8eaf75d\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " Oct 30 06:17:42.584342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.583398 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-kube-api-access-fv626\" (UniqueName: \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-kube-api-access-fv626\") pod \"1a443ba6-2255-490b-90b2-4922b8eaf75d\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " Oct 30 06:17:42.584342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.583428 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-socket\") pod \"1a443ba6-2255-490b-90b2-4922b8eaf75d\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " Oct 30 06:17:42.584342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.583458 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-token\" (UniqueName: \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-token\") pod \"1a443ba6-2255-490b-90b2-4922b8eaf75d\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " Oct 30 06:17:42.584342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.583518 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-podinfo\") pod \"1a443ba6-2255-490b-90b2-4922b8eaf75d\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " Oct 30 06:17:42.584342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.583588 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a443ba6-2255-490b-90b2-4922b8eaf75d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a443ba6-2255-490b-90b2-4922b8eaf75d-istiod-ca-cert\") pod \"1a443ba6-2255-490b-90b2-4922b8eaf75d\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " Oct 30 06:17:42.584342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.583617 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-data\") pod \"1a443ba6-2255-490b-90b2-4922b8eaf75d\" (UID: \"1a443ba6-2255-490b-90b2-4922b8eaf75d\") " Oct 30 06:17:42.604339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.601776 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-data" pod "1a443ba6-2255-490b-90b2-4922b8eaf75d" (UID: "1a443ba6-2255-490b-90b2-4922b8eaf75d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.604339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.601990 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-socket" pod "1a443ba6-2255-490b-90b2-4922b8eaf75d" (UID: "1a443ba6-2255-490b-90b2-4922b8eaf75d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.616178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.609511 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-certs" pod "1a443ba6-2255-490b-90b2-4922b8eaf75d" (UID: "1a443ba6-2255-490b-90b2-4922b8eaf75d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.616178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.619632 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a443ba6-2255-490b-90b2-4922b8eaf75d-istiod-ca-cert" pod "1a443ba6-2255-490b-90b2-4922b8eaf75d" (UID: "1a443ba6-2255-490b-90b2-4922b8eaf75d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:42.635788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.628001 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-podinfo" pod "1a443ba6-2255-490b-90b2-4922b8eaf75d" (UID: "1a443ba6-2255-490b-90b2-4922b8eaf75d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:42.636790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.636734 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-kube-api-access-fv626" pod "1a443ba6-2255-490b-90b2-4922b8eaf75d" (UID: "1a443ba6-2255-490b-90b2-4922b8eaf75d"). InnerVolumeSpecName "kube-api-access-fv626". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:42.640608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.640554 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-token" pod "1a443ba6-2255-490b-90b2-4922b8eaf75d" (UID: "1a443ba6-2255-490b-90b2-4922b8eaf75d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:42.668343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.648328 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-credential-socket" pod "1a443ba6-2255-490b-90b2-4922b8eaf75d" (UID: "1a443ba6-2255-490b-90b2-4922b8eaf75d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.733794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.732864 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.734023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.734004 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv626\" (UniqueName: \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-kube-api-access-fv626\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.734176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.734162 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.734386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.734284 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.734519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.734503 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.734665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.734651 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a443ba6-2255-490b-90b2-4922b8eaf75d-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.734783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.734771 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.819816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.819405 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:17:42.819816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.819459 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:17:42.836259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.836225 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.836691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.836647 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-envoy" pod "1a443ba6-2255-490b-90b2-4922b8eaf75d" (UID: "1a443ba6-2255-490b-90b2-4922b8eaf75d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.900671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.900639 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:17:42.906142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.906065 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.942902 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-token\" (UniqueName: \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-token\") pod \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.942985 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-certs\") pod \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943029 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-data\") pod \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943056 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-certs\") pod \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943092 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-podinfo\") pod \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943141 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-credential-socket\") pod \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943186 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-kube-api-access-b4wxw\" (UniqueName: \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-kube-api-access-b4wxw\") pod \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943237 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-envoy\") pod \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943282 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-token\" (UniqueName: \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-token\") pod \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943332 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-kube-api-access-f9wzg\" (UniqueName: \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-kube-api-access-f9wzg\") pod \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943434 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istiod-ca-cert\") pod \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943474 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-socket\") pod \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943558 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-envoy\") pod \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943652 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-socket\") pod \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943701 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-podinfo\") pod \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " Oct 30 06:17:42.943831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.943760 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-credential-socket\") pod \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " Oct 30 06:17:42.946759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.944931 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-data\") pod \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\" (UID: \"43cec6bc-bd19-4c3b-9520-8fd3eb41c005\") " Oct 30 06:17:42.946759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.945022 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istiod-ca-cert\") pod \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\" (UID: \"9d1d0f14-43a9-4609-a7c7-de84d0c47758\") " Oct 30 06:17:42.946759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.945378 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-credential-socket" pod "9d1d0f14-43a9-4609-a7c7-de84d0c47758" (UID: "9d1d0f14-43a9-4609-a7c7-de84d0c47758"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.946759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.945615 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-socket" pod "43cec6bc-bd19-4c3b-9520-8fd3eb41c005" (UID: "43cec6bc-bd19-4c3b-9520-8fd3eb41c005"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.947655 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.947485 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-data" pod "43cec6bc-bd19-4c3b-9520-8fd3eb41c005" (UID: "43cec6bc-bd19-4c3b-9520-8fd3eb41c005"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.948493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.947888 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istiod-ca-cert" pod "9d1d0f14-43a9-4609-a7c7-de84d0c47758" (UID: "9d1d0f14-43a9-4609-a7c7-de84d0c47758"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:42.948493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.948323 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-certs" pod "9d1d0f14-43a9-4609-a7c7-de84d0c47758" (UID: "9d1d0f14-43a9-4609-a7c7-de84d0c47758"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.949375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.948960 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-credential-socket" pod "43cec6bc-bd19-4c3b-9520-8fd3eb41c005" (UID: "43cec6bc-bd19-4c3b-9520-8fd3eb41c005"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.949375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.949001 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a443ba6-2255-490b-90b2-4922b8eaf75d-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.949375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.949025 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.949375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.949039 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.949375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.949053 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.949375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.949068 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.949375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.949081 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.949821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.949391 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istiod-ca-cert" pod "43cec6bc-bd19-4c3b-9520-8fd3eb41c005" (UID: "43cec6bc-bd19-4c3b-9520-8fd3eb41c005"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:42.950130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.949932 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-certs" pod "43cec6bc-bd19-4c3b-9520-8fd3eb41c005" (UID: "43cec6bc-bd19-4c3b-9520-8fd3eb41c005"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.950211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.950165 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-data" pod "9d1d0f14-43a9-4609-a7c7-de84d0c47758" (UID: "9d1d0f14-43a9-4609-a7c7-de84d0c47758"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.950581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.950366 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-socket" pod "9d1d0f14-43a9-4609-a7c7-de84d0c47758" (UID: "9d1d0f14-43a9-4609-a7c7-de84d0c47758"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:42.992290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.974492 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-token" pod "9d1d0f14-43a9-4609-a7c7-de84d0c47758" (UID: "9d1d0f14-43a9-4609-a7c7-de84d0c47758"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:43.023681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.999007 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-envoy" pod "9d1d0f14-43a9-4609-a7c7-de84d0c47758" (UID: "9d1d0f14-43a9-4609-a7c7-de84d0c47758"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:43.023681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.999023 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-podinfo" pod "43cec6bc-bd19-4c3b-9520-8fd3eb41c005" (UID: "43cec6bc-bd19-4c3b-9520-8fd3eb41c005"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:43.023681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.999049 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-kube-api-access-b4wxw" pod "9d1d0f14-43a9-4609-a7c7-de84d0c47758" (UID: "9d1d0f14-43a9-4609-a7c7-de84d0c47758"). InnerVolumeSpecName "kube-api-access-b4wxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:43.023681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.999124 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-envoy" pod "43cec6bc-bd19-4c3b-9520-8fd3eb41c005" (UID: "43cec6bc-bd19-4c3b-9520-8fd3eb41c005"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:43.023681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.999143 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-podinfo" pod "9d1d0f14-43a9-4609-a7c7-de84d0c47758" (UID: "9d1d0f14-43a9-4609-a7c7-de84d0c47758"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:43.023681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.999173 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-token" pod "43cec6bc-bd19-4c3b-9520-8fd3eb41c005" (UID: "43cec6bc-bd19-4c3b-9520-8fd3eb41c005"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:43.023681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:42.999249 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-kube-api-access-f9wzg" pod "43cec6bc-bd19-4c3b-9520-8fd3eb41c005" (UID: "43cec6bc-bd19-4c3b-9520-8fd3eb41c005"). InnerVolumeSpecName "kube-api-access-f9wzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:43.035331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.033055 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:17:43.045128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.045093 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz"] Oct 30 06:17:43.046262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.046240 5001 cpu_manager.go:414] "containerMap: removing container" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047443 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047500 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047509 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047548 5001 cpu_manager.go:414] "containerMap: removing container" podUID="63eca24a-c41d-4d53-b691-98d71d929310" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047556 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63eca24a-c41d-4d53-b691-98d71d929310" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047579 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047587 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047636 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047643 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047660 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047667 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047857 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047870 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047885 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047895 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047913 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="63eca24a-c41d-4d53-b691-98d71d929310" containerName="istio-proxy" Oct 30 06:17:43.048623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.047924 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" containerName="istio-proxy" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051103 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051735 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051771 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051786 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051798 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051812 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051826 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051840 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4wxw\" (UniqueName: \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-kube-api-access-b4wxw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051853 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051867 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051879 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9wzg\" (UniqueName: \"kubernetes.io/projected/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-kube-api-access-f9wzg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051892 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cec6bc-bd19-4c3b-9520-8fd3eb41c005-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051915 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.055335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.051927 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1d0f14-43a9-4609-a7c7-de84d0c47758-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.095103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.090718 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-31407e\"" Oct 30 06:17:43.095103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.090836 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-31407e-dockercfg-dzshj\"" Oct 30 06:17:43.126074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.124320 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz"] Oct 30 06:17:43.133513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.133478 5001 generic.go:356] "Generic (PLEG): container finished" podID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerID="8664586e94dd95a6a6bec68f7173ef732a675683eff5112a4e4394f3089ed6fb" exitCode=0 Oct 30 06:17:43.133688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.133609 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" event={"ID":"56ed7d68-efcb-424b-8289-ad2c278768c2","Type":"ContainerDied","Data":"8664586e94dd95a6a6bec68f7173ef732a675683eff5112a4e4394f3089ed6fb"} Oct 30 06:17:43.136127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.136097 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" event={"ID":"1a443ba6-2255-490b-90b2-4922b8eaf75d","Type":"ContainerDied","Data":"574b8b1ed2f9cd1374b373f87b6cea7134489faacfae8b1c5cfdc9a0e3a9d52a"} Oct 30 06:17:43.136268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.136154 5001 scope.go:122] "RemoveContainer" containerID="57f5e65e93e3fef5e23228f22ccb786a58bd46f2f5c3d58d71c5f26bfead1d93" Oct 30 06:17:43.136344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.136317 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz" Oct 30 06:17:43.149256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.149186 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" event={"ID":"43cec6bc-bd19-4c3b-9520-8fd3eb41c005","Type":"ContainerDied","Data":"da206877c07f9a6a5cd79ce533fbd5d0cf62a72d5f9d5ae8e40066b742cfba0e"} Oct 30 06:17:43.149661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.149643 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz" Oct 30 06:17:43.161440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.154518 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-socket\") pod \"63eca24a-c41d-4d53-b691-98d71d929310\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " Oct 30 06:17:43.161440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.154585 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-credential-socket\") pod \"63eca24a-c41d-4d53-b691-98d71d929310\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " Oct 30 06:17:43.161440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.154711 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-envoy\") pod \"63eca24a-c41d-4d53-b691-98d71d929310\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " Oct 30 06:17:43.161440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.154761 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-data\") pod \"63eca24a-c41d-4d53-b691-98d71d929310\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " Oct 30 06:17:43.161440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.154789 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-istio-token\" (UniqueName: \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-istio-token\") pod \"63eca24a-c41d-4d53-b691-98d71d929310\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " Oct 30 06:17:43.161440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.154821 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63eca24a-c41d-4d53-b691-98d71d929310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63eca24a-c41d-4d53-b691-98d71d929310-istio-podinfo\") pod \"63eca24a-c41d-4d53-b691-98d71d929310\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " Oct 30 06:17:43.161440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.154861 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-certs\") pod \"63eca24a-c41d-4d53-b691-98d71d929310\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " Oct 30 06:17:43.161440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.154959 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63eca24a-c41d-4d53-b691-98d71d929310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63eca24a-c41d-4d53-b691-98d71d929310-istiod-ca-cert\") pod \"63eca24a-c41d-4d53-b691-98d71d929310\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " Oct 30 06:17:43.161440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.155047 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-kube-api-access-qff6f\" (UniqueName: \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-kube-api-access-qff6f\") pod \"63eca24a-c41d-4d53-b691-98d71d929310\" (UID: \"63eca24a-c41d-4d53-b691-98d71d929310\") " Oct 30 06:17:43.161440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.155453 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3d7158-7803-403a-b160-da80e0d7a684-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz\" (UID: \"ce3d7158-7803-403a-b160-da80e0d7a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:17:43.161440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.155685 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r27z\" (UniqueName: \"kubernetes.io/projected/ce3d7158-7803-403a-b160-da80e0d7a684-kube-api-access-5r27z\") pod \"payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz\" (UID: \"ce3d7158-7803-403a-b160-da80e0d7a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:17:43.164837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.164184 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-certs" pod "63eca24a-c41d-4d53-b691-98d71d929310" (UID: "63eca24a-c41d-4d53-b691-98d71d929310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:43.164837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.164574 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63eca24a-c41d-4d53-b691-98d71d929310-istiod-ca-cert" pod "63eca24a-c41d-4d53-b691-98d71d929310" (UID: "63eca24a-c41d-4d53-b691-98d71d929310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:43.194346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.168279 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-data" pod "63eca24a-c41d-4d53-b691-98d71d929310" (UID: "63eca24a-c41d-4d53-b691-98d71d929310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:43.194346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.171500 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-credential-socket" pod "63eca24a-c41d-4d53-b691-98d71d929310" (UID: "63eca24a-c41d-4d53-b691-98d71d929310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:43.194346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.183482 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-socket" pod "63eca24a-c41d-4d53-b691-98d71d929310" (UID: "63eca24a-c41d-4d53-b691-98d71d929310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:43.194346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.185382 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" event={"ID":"63eca24a-c41d-4d53-b691-98d71d929310","Type":"ContainerDied","Data":"12f347f4d4cec1193b9600b159f30176b8202fdb43c77bbca6f5f7755f5d383e"} Oct 30 06:17:43.194346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.185508 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2" Oct 30 06:17:43.202758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.202723 5001 scope.go:122] "RemoveContainer" containerID="f97bcc009475f5b6960ac91e1838404d5943c34f480948f009226207e1050c60" Oct 30 06:17:43.210586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.210551 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" event={"ID":"9d1d0f14-43a9-4609-a7c7-de84d0c47758","Type":"ContainerDied","Data":"4acc464bfdd4718fff74df3fbc7417b333dc58e2f28e4d384755aa8d06b9582c"} Oct 30 06:17:43.210711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.210692 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g" Oct 30 06:17:43.226114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.225860 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" event={"ID":"5da87567-9dee-47ec-9bfd-52f7cb332923","Type":"ContainerStarted","Data":"0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f"} Oct 30 06:17:43.227546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.227493 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-kube-api-access-qff6f" pod "63eca24a-c41d-4d53-b691-98d71d929310" (UID: "63eca24a-c41d-4d53-b691-98d71d929310"). InnerVolumeSpecName "kube-api-access-qff6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:43.227800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.227761 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-envoy" pod "63eca24a-c41d-4d53-b691-98d71d929310" (UID: "63eca24a-c41d-4d53-b691-98d71d929310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:43.229072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.228281 5001 scope.go:122] "RemoveContainer" containerID="2352bf41ac2e3e5cf43c05d395a8ffe99ce9a2d2611e487abf914879e3627df3" Oct 30 06:17:43.230252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.230107 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-istio-token" pod "63eca24a-c41d-4d53-b691-98d71d929310" (UID: "63eca24a-c41d-4d53-b691-98d71d929310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:43.230546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.230504 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63eca24a-c41d-4d53-b691-98d71d929310-istio-podinfo" pod "63eca24a-c41d-4d53-b691-98d71d929310" (UID: "63eca24a-c41d-4d53-b691-98d71d929310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:43.244349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.244330 5001 scope.go:122] "RemoveContainer" containerID="32200783d1486cc072fe792cadf3ed115fc012ca07faa87b47b30c0c0569a3db" Oct 30 06:17:43.282156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.281118 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3d7158-7803-403a-b160-da80e0d7a684-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz\" (UID: \"ce3d7158-7803-403a-b160-da80e0d7a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:17:43.282156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.281500 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r27z\" (UniqueName: \"kubernetes.io/projected/ce3d7158-7803-403a-b160-da80e0d7a684-kube-api-access-5r27z\") pod \"payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz\" (UID: \"ce3d7158-7803-403a-b160-da80e0d7a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:17:43.282156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.281598 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63eca24a-c41d-4d53-b691-98d71d929310-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.282156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.281617 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qff6f\" (UniqueName: \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-kube-api-access-qff6f\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.282156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.281636 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.282156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.281650 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.282156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.281663 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.282156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.281675 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.282156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.281688 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63eca24a-c41d-4d53-b691-98d71d929310-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.282156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.281700 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63eca24a-c41d-4d53-b691-98d71d929310-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.282156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.281712 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63eca24a-c41d-4d53-b691-98d71d929310-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:43.284346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.284297 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3d7158-7803-403a-b160-da80e0d7a684-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz\" (UID: \"ce3d7158-7803-403a-b160-da80e0d7a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:17:43.613560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.605375 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz"] Oct 30 06:17:43.613560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.605449 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-ck2vz"] Oct 30 06:17:43.657599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.646255 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:17:43.755266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.755235 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:17:43.755885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.755850 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:17:43.875487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.821464 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:17:43.875487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.821526 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:17:43.875487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.439245 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r27z\" (UniqueName: \"kubernetes.io/projected/ce3d7158-7803-403a-b160-da80e0d7a684-kube-api-access-5r27z\") pod \"payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz\" (UID: \"ce3d7158-7803-403a-b160-da80e0d7a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:17:43.875487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.863212 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz"] Oct 30 06:17:43.929848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.929586 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-wqhzz"] Oct 30 06:17:43.950579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:43.946783 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km"] Oct 30 06:17:44.020514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.020487 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g"] Oct 30 06:17:44.032433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.032399 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zxv5g"] Oct 30 06:17:44.032667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.026492 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.042119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.026361 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-certs\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.042119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.040898 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istiod-ca-cert\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.042119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.040952 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-data\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.042119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.040987 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-podinfo\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.042119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.041026 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-token\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.047133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.041197 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfcz\" (UniqueName: \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-kube-api-access-4cfcz\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.047133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.046728 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.047133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.046834 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-credential-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.047133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.046861 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-envoy\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.048216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.047720 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km"] Oct 30 06:17:44.048216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.047981 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:17:44.069247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.069211 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-532f3b-openshift-default-dockercfg-hfslq\"" Oct 30 06:17:44.150549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.150505 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-podinfo\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.150720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.150565 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-token\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.150720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.150712 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfcz\" (UniqueName: \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-kube-api-access-4cfcz\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.150839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.150743 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.150839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.150821 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-credential-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.150941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.150849 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-envoy\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.150941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.150933 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-certs\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.151098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.151066 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istiod-ca-cert\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.151161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.151103 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-data\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.185483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.157567 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.185483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.159431 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-certs\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.185483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.159994 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istiod-ca-cert\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.185483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.160207 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-credential-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.185483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.168557 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a443ba6-2255-490b-90b2-4922b8eaf75d" path="/var/lib/kubelet/pods/1a443ba6-2255-490b-90b2-4922b8eaf75d/volumes" Oct 30 06:17:44.185483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.169199 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3a428c-c55f-4ea6-b9e2-be1dce741eaf" path="/var/lib/kubelet/pods/2c3a428c-c55f-4ea6-b9e2-be1dce741eaf/volumes" Oct 30 06:17:44.185483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.169868 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43cec6bc-bd19-4c3b-9520-8fd3eb41c005" path="/var/lib/kubelet/pods/43cec6bc-bd19-4c3b-9520-8fd3eb41c005/volumes" Oct 30 06:17:44.185483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.170958 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1d0f14-43a9-4609-a7c7-de84d0c47758" path="/var/lib/kubelet/pods/9d1d0f14-43a9-4609-a7c7-de84d0c47758/volumes" Oct 30 06:17:44.185483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.171418 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2"] Oct 30 06:17:44.185483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.171457 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9gdd2"] Oct 30 06:17:44.186731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.186704 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-data\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.189946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.189705 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-envoy\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.266468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.248191 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-token\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.333702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.297249 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfcz\" (UniqueName: \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-kube-api-access-4cfcz\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.333702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.331348 5001 generic.go:356] "Generic (PLEG): container finished" podID="ad5ce104-9403-4712-8512-9845efc5ac51" containerID="7381bc7f6f15ac882500104dd3fb6dee705cc44f91cd5b3ce31950ef53a8b41b" exitCode=0 Oct 30 06:17:44.333702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.331522 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" event={"ID":"ad5ce104-9403-4712-8512-9845efc5ac51","Type":"ContainerDied","Data":"7381bc7f6f15ac882500104dd3fb6dee705cc44f91cd5b3ce31950ef53a8b41b"} Oct 30 06:17:44.337499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.337353 5001 generic.go:356] "Generic (PLEG): container finished" podID="1bcadd15-1201-491a-8348-dd39b94f9bae" containerID="15ce3dc383cb1eb254b2aed7d792942023c02235cfb13dff6491c01b1abfeeac" exitCode=0 Oct 30 06:17:44.337499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.337481 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" event={"ID":"1bcadd15-1201-491a-8348-dd39b94f9bae","Type":"ContainerDied","Data":"15ce3dc383cb1eb254b2aed7d792942023c02235cfb13dff6491c01b1abfeeac"} Oct 30 06:17:44.345368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.345290 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-podinfo\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.459199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.458879 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:44.621641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.591414 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:17:44.621641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.591479 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:17:44.780872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.780780 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:17:44.781478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.781444 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:17:44.873401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:44.851643 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:17:45.036861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.036787 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.209272 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-podinfo\") pod \"56ed7d68-efcb-424b-8289-ad2c278768c2\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.209438 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-socket\") pod \"56ed7d68-efcb-424b-8289-ad2c278768c2\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.209582 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-certs\") pod \"56ed7d68-efcb-424b-8289-ad2c278768c2\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.209619 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-credential-socket\") pod \"56ed7d68-efcb-424b-8289-ad2c278768c2\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.209698 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-envoy\") pod \"56ed7d68-efcb-424b-8289-ad2c278768c2\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.209743 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-kube-api-access-d4vmb\" (UniqueName: \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-kube-api-access-d4vmb\") pod \"56ed7d68-efcb-424b-8289-ad2c278768c2\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.209860 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ed7d68-efcb-424b-8289-ad2c278768c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ed7d68-efcb-424b-8289-ad2c278768c2-istiod-ca-cert\") pod \"56ed7d68-efcb-424b-8289-ad2c278768c2\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.209898 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-token\" (UniqueName: \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-token\") pod \"56ed7d68-efcb-424b-8289-ad2c278768c2\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.209962 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-data\") pod \"56ed7d68-efcb-424b-8289-ad2c278768c2\" (UID: \"56ed7d68-efcb-424b-8289-ad2c278768c2\") " Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.215665 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-credential-socket" pod "56ed7d68-efcb-424b-8289-ad2c278768c2" (UID: "56ed7d68-efcb-424b-8289-ad2c278768c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.216392 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-socket" pod "56ed7d68-efcb-424b-8289-ad2c278768c2" (UID: "56ed7d68-efcb-424b-8289-ad2c278768c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.216832 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-data" pod "56ed7d68-efcb-424b-8289-ad2c278768c2" (UID: "56ed7d68-efcb-424b-8289-ad2c278768c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.217176 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ed7d68-efcb-424b-8289-ad2c278768c2-istiod-ca-cert" pod "56ed7d68-efcb-424b-8289-ad2c278768c2" (UID: "56ed7d68-efcb-424b-8289-ad2c278768c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:45.218466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.217801 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-certs" pod "56ed7d68-efcb-424b-8289-ad2c278768c2" (UID: "56ed7d68-efcb-424b-8289-ad2c278768c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.226199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.222805 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:17:45.226199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.223711 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-podinfo" pod "56ed7d68-efcb-424b-8289-ad2c278768c2" (UID: "56ed7d68-efcb-424b-8289-ad2c278768c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:45.226199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.226146 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-kube-api-access-d4vmb" pod "56ed7d68-efcb-424b-8289-ad2c278768c2" (UID: "56ed7d68-efcb-424b-8289-ad2c278768c2"). InnerVolumeSpecName "kube-api-access-d4vmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.314406 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-credential-socket\") pod \"ad5ce104-9403-4712-8512-9845efc5ac51\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.314564 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-data\") pod \"ad5ce104-9403-4712-8512-9845efc5ac51\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.314654 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-istio-token\") pod \"ad5ce104-9403-4712-8512-9845efc5ac51\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.314702 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-envoy\") pod \"ad5ce104-9403-4712-8512-9845efc5ac51\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.314730 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad5ce104-9403-4712-8512-9845efc5ac51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ce104-9403-4712-8512-9845efc5ac51-istio-podinfo\") pod \"ad5ce104-9403-4712-8512-9845efc5ac51\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.314756 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5ce104-9403-4712-8512-9845efc5ac51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ce104-9403-4712-8512-9845efc5ac51-istiod-ca-cert\") pod \"ad5ce104-9403-4712-8512-9845efc5ac51\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.314783 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-certs\") pod \"ad5ce104-9403-4712-8512-9845efc5ac51\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.314847 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-kube-api-access-xlrln\" (UniqueName: \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-kube-api-access-xlrln\") pod \"ad5ce104-9403-4712-8512-9845efc5ac51\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.314964 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-socket\") pod \"ad5ce104-9403-4712-8512-9845efc5ac51\" (UID: \"ad5ce104-9403-4712-8512-9845efc5ac51\") " Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.315847 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vmb\" (UniqueName: \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-kube-api-access-d4vmb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.315885 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ed7d68-efcb-424b-8289-ad2c278768c2-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.315903 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.315942 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.315957 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.315981 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.345883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.315998 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.366726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.357631 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-data" pod "ad5ce104-9403-4712-8512-9845efc5ac51" (UID: "ad5ce104-9403-4712-8512-9845efc5ac51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.366726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.357881 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-credential-socket" pod "ad5ce104-9403-4712-8512-9845efc5ac51" (UID: "ad5ce104-9403-4712-8512-9845efc5ac51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.366726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.358957 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-socket" pod "ad5ce104-9403-4712-8512-9845efc5ac51" (UID: "ad5ce104-9403-4712-8512-9845efc5ac51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.366726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.359897 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5ce104-9403-4712-8512-9845efc5ac51-istiod-ca-cert" pod "ad5ce104-9403-4712-8512-9845efc5ac51" (UID: "ad5ce104-9403-4712-8512-9845efc5ac51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:45.366726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.365867 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-envoy" pod "56ed7d68-efcb-424b-8289-ad2c278768c2" (UID: "56ed7d68-efcb-424b-8289-ad2c278768c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.366726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.366687 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-certs" pod "ad5ce104-9403-4712-8512-9845efc5ac51" (UID: "ad5ce104-9403-4712-8512-9845efc5ac51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.430331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.427049 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.430331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.427085 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.430331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.427100 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ce104-9403-4712-8512-9845efc5ac51-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.430331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.427114 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.430331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.427128 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.430331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.427141 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.497125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.497086 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" event={"ID":"56ed7d68-efcb-424b-8289-ad2c278768c2","Type":"ContainerDied","Data":"63814e3160f56700e4c588df4fa3caf8323c6174d205575612cbfff8d712d876"} Oct 30 06:17:45.497455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.497421 5001 scope.go:122] "RemoveContainer" containerID="8664586e94dd95a6a6bec68f7173ef732a675683eff5112a4e4394f3089ed6fb" Oct 30 06:17:45.497816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.497785 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j" Oct 30 06:17:45.621452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.618569 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-istio-token" pod "ad5ce104-9403-4712-8512-9845efc5ac51" (UID: "ad5ce104-9403-4712-8512-9845efc5ac51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:45.621452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.618677 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad5ce104-9403-4712-8512-9845efc5ac51-istio-podinfo" pod "ad5ce104-9403-4712-8512-9845efc5ac51" (UID: "ad5ce104-9403-4712-8512-9845efc5ac51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:45.621452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.618745 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-kube-api-access-xlrln" pod "ad5ce104-9403-4712-8512-9845efc5ac51" (UID: "ad5ce104-9403-4712-8512-9845efc5ac51"). InnerVolumeSpecName "kube-api-access-xlrln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:45.621452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.618799 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-envoy" pod "ad5ce104-9403-4712-8512-9845efc5ac51" (UID: "ad5ce104-9403-4712-8512-9845efc5ac51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.621452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.619017 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-token" pod "56ed7d68-efcb-424b-8289-ad2c278768c2" (UID: "56ed7d68-efcb-424b-8289-ad2c278768c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:45.621452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.620993 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" event={"ID":"ad5ce104-9403-4712-8512-9845efc5ac51","Type":"ContainerDied","Data":"ce22139329e95663a5dc5b5825bb5e71fec16016224d531883d279fdd65482e6"} Oct 30 06:17:45.621452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.621111 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" Oct 30 06:17:45.621992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.621502 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:17:45.621992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.621583 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:17:45.634915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.634892 5001 scope.go:122] "RemoveContainer" containerID="7381bc7f6f15ac882500104dd3fb6dee705cc44f91cd5b3ce31950ef53a8b41b" Oct 30 06:17:45.645200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.645168 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz"] Oct 30 06:17:45.647642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.646081 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ed7d68-efcb-424b-8289-ad2c278768c2-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.647642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.646841 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.647642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.646854 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ce104-9403-4712-8512-9845efc5ac51-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.647642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.646868 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ce104-9403-4712-8512-9845efc5ac51-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.647642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.646880 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlrln\" (UniqueName: \"kubernetes.io/projected/ad5ce104-9403-4712-8512-9845efc5ac51-kube-api-access-xlrln\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.647642 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:17:45.647558 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3d7158_7803_403a_b160_da80e0d7a684.slice/crio-381651531cf0dd6d377ca9b8e1e3233b4efbb022b4a4ec63f46d7bf3303f17c8 WatchSource:0}: Error finding container 381651531cf0dd6d377ca9b8e1e3233b4efbb022b4a4ec63f46d7bf3303f17c8: Status 404 returned error can't find the container with id 381651531cf0dd6d377ca9b8e1e3233b4efbb022b4a4ec63f46d7bf3303f17c8 Oct 30 06:17:45.731003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.730103 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:17:45.736790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.736742 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km"] Oct 30 06:17:45.740944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.740888 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:45.741074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.740977 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:45.741074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.741020 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:45.802345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.789514 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:17:45.802345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.789573 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:17:45.823780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.823704 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2"] Oct 30 06:17:45.829546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.829510 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2"] Oct 30 06:17:45.863937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.862374 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-credential-socket\") pod \"1bcadd15-1201-491a-8348-dd39b94f9bae\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " Oct 30 06:17:45.863937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.862467 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-certs\") pod \"1bcadd15-1201-491a-8348-dd39b94f9bae\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " Oct 30 06:17:45.863937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.862566 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-socket\") pod \"1bcadd15-1201-491a-8348-dd39b94f9bae\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " Oct 30 06:17:45.863937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.862628 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-envoy\") pod \"1bcadd15-1201-491a-8348-dd39b94f9bae\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " Oct 30 06:17:45.863937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.862666 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-kube-api-access-697x6\" (UniqueName: \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-kube-api-access-697x6\") pod \"1bcadd15-1201-491a-8348-dd39b94f9bae\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " Oct 30 06:17:45.863937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.862716 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-token\" (UniqueName: \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-token\") pod \"1bcadd15-1201-491a-8348-dd39b94f9bae\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " Oct 30 06:17:45.863937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.862795 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bcadd15-1201-491a-8348-dd39b94f9bae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcadd15-1201-491a-8348-dd39b94f9bae-istiod-ca-cert\") pod \"1bcadd15-1201-491a-8348-dd39b94f9bae\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " Oct 30 06:17:45.863937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.862888 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-data\") pod \"1bcadd15-1201-491a-8348-dd39b94f9bae\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " Oct 30 06:17:45.863937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.862937 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-podinfo\") pod \"1bcadd15-1201-491a-8348-dd39b94f9bae\" (UID: \"1bcadd15-1201-491a-8348-dd39b94f9bae\") " Oct 30 06:17:45.873512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.873045 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-certs" pod "1bcadd15-1201-491a-8348-dd39b94f9bae" (UID: "1bcadd15-1201-491a-8348-dd39b94f9bae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.873512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.873203 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-credential-socket" pod "1bcadd15-1201-491a-8348-dd39b94f9bae" (UID: "1bcadd15-1201-491a-8348-dd39b94f9bae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.873512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.873361 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-socket" pod "1bcadd15-1201-491a-8348-dd39b94f9bae" (UID: "1bcadd15-1201-491a-8348-dd39b94f9bae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.874322 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.874074 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bcadd15-1201-491a-8348-dd39b94f9bae-istiod-ca-cert" pod "1bcadd15-1201-491a-8348-dd39b94f9bae" (UID: "1bcadd15-1201-491a-8348-dd39b94f9bae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:45.874532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.874493 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-data" pod "1bcadd15-1201-491a-8348-dd39b94f9bae" (UID: "1bcadd15-1201-491a-8348-dd39b94f9bae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.912846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.912808 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-token" pod "1bcadd15-1201-491a-8348-dd39b94f9bae" (UID: "1bcadd15-1201-491a-8348-dd39b94f9bae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:45.918448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.918378 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-podinfo" pod "1bcadd15-1201-491a-8348-dd39b94f9bae" (UID: "1bcadd15-1201-491a-8348-dd39b94f9bae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:45.919151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.919106 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-kube-api-access-697x6" pod "1bcadd15-1201-491a-8348-dd39b94f9bae" (UID: "1bcadd15-1201-491a-8348-dd39b94f9bae"). InnerVolumeSpecName "kube-api-access-697x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:45.927460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.927423 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-envoy" pod "1bcadd15-1201-491a-8348-dd39b94f9bae" (UID: "1bcadd15-1201-491a-8348-dd39b94f9bae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:45.939263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.939238 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j"] Oct 30 06:17:45.946704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.946673 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hqw9j"] Oct 30 06:17:45.982945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.982903 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.986386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.986359 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.995398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.994361 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.995398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.994402 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.995398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.994419 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.995398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.994431 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.995398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.994448 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-697x6\" (UniqueName: \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-kube-api-access-697x6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.995398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.994461 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcadd15-1201-491a-8348-dd39b94f9bae-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:45.995398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:45.994474 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcadd15-1201-491a-8348-dd39b94f9bae-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:46.044058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.042620 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:17:46.044058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.042731 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cf8t2" podUID="ad5ce104-9403-4712-8512-9845efc5ac51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:17:46.139379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.139224 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" path="/var/lib/kubelet/pods/56ed7d68-efcb-424b-8289-ad2c278768c2/volumes" Oct 30 06:17:46.142341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.139994 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63eca24a-c41d-4d53-b691-98d71d929310" path="/var/lib/kubelet/pods/63eca24a-c41d-4d53-b691-98d71d929310/volumes" Oct 30 06:17:46.142341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.140636 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5ce104-9403-4712-8512-9845efc5ac51" path="/var/lib/kubelet/pods/ad5ce104-9403-4712-8512-9845efc5ac51/volumes" Oct 30 06:17:46.595396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.594898 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:17:46.595396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.594957 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:17:46.634580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.633724 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" event={"ID":"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1","Type":"ContainerStarted","Data":"52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9"} Oct 30 06:17:46.634580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.633779 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" event={"ID":"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1","Type":"ContainerStarted","Data":"45cdf9a4ebc81bc2d0a5f5af91a3e4a70e6a2ac9353f4b006916b36024ce03b2"} Oct 30 06:17:46.637615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.637085 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" event={"ID":"ce3d7158-7803-403a-b160-da80e0d7a684","Type":"ContainerStarted","Data":"86a42b51b3021d2b08b8460cd17e2a59cc002a4cdadb44476a48e4f831c7ea3f"} Oct 30 06:17:46.637615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.637126 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" event={"ID":"ce3d7158-7803-403a-b160-da80e0d7a684","Type":"ContainerStarted","Data":"381651531cf0dd6d377ca9b8e1e3233b4efbb022b4a4ec63f46d7bf3303f17c8"} Oct 30 06:17:46.637926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.637883 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:17:46.640741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.640672 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" event={"ID":"1bcadd15-1201-491a-8348-dd39b94f9bae","Type":"ContainerDied","Data":"41691aaf33d95975bd1096f6937a8dec355936ddc8c49ae298e59dcd9cd331ac"} Oct 30 06:17:46.640741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.640722 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn" Oct 30 06:17:46.640868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.640724 5001 scope.go:122] "RemoveContainer" containerID="15ce3dc383cb1eb254b2aed7d792942023c02235cfb13dff6491c01b1abfeeac" Oct 30 06:17:46.726331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.701101 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" podStartSLOduration=3.701080267 podStartE2EDuration="3.701080267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:46.684636357 +0000 UTC m=+1480.680593368" watchObservedRunningTime="2026-10-30 06:17:46.701080267 +0000 UTC m=+1480.697037278" Oct 30 06:17:46.747683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.747632 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" podStartSLOduration=4.747616996 podStartE2EDuration="4.747616996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:46.726836686 +0000 UTC m=+1480.722793698" watchObservedRunningTime="2026-10-30 06:17:46.747616996 +0000 UTC m=+1480.743574045" Oct 30 06:17:46.788335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.787456 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:17:46.788335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.787525 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:17:46.805853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.803394 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn"] Oct 30 06:17:46.818343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.808725 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-jtrmn"] Oct 30 06:17:46.876328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:46.872463 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:17:47.520404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.519793 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:47.526278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.526251 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:47.587385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.586822 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:17:47.587385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.586885 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:17:47.724583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.723728 5001 generic.go:356] "Generic (PLEG): container finished" podID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerID="6e62ce5bdcd3faefa71532b83323531c5d0d9251562db1a2df7cf641e4e919ae" exitCode=0 Oct 30 06:17:47.724583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.723850 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" event={"ID":"bf6842a8-36df-4ea9-8639-a7a7cb18e577","Type":"ContainerDied","Data":"6e62ce5bdcd3faefa71532b83323531c5d0d9251562db1a2df7cf641e4e919ae"} Oct 30 06:17:47.724583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.723936 5001 scope.go:122] "RemoveContainer" containerID="db541d13c4fdd3522aec928d0985373f7374bf475fa416591adb849b12897636" Oct 30 06:17:47.729173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.729119 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:47.730878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.730771 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:17:47.800470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.799447 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:17:47.800470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.799507 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:17:47.855583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:47.855555 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:17:48.013186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.013145 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-token\" (UniqueName: \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-token\") pod \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " Oct 30 06:17:48.013386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.013199 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-kube-api-access-ggnqp\" (UniqueName: \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-kube-api-access-ggnqp\") pod \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " Oct 30 06:17:48.013386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.013239 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-envoy\") pod \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " Oct 30 06:17:48.013386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.013365 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istiod-ca-cert\") pod \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " Oct 30 06:17:48.013582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.013415 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-socket\") pod \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " Oct 30 06:17:48.013653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.013637 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-credential-socket\") pod \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " Oct 30 06:17:48.013707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.013693 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-data\") pod \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " Oct 30 06:17:48.013755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.013729 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-certs\") pod \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " Oct 30 06:17:48.013788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.013774 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-podinfo\") pod \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\" (UID: \"bf6842a8-36df-4ea9-8639-a7a7cb18e577\") " Oct 30 06:17:48.015062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.015036 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istiod-ca-cert" pod "bf6842a8-36df-4ea9-8639-a7a7cb18e577" (UID: "bf6842a8-36df-4ea9-8639-a7a7cb18e577"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:48.015653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.015620 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-socket" pod "bf6842a8-36df-4ea9-8639-a7a7cb18e577" (UID: "bf6842a8-36df-4ea9-8639-a7a7cb18e577"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:48.015949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.015813 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-credential-socket" pod "bf6842a8-36df-4ea9-8639-a7a7cb18e577" (UID: "bf6842a8-36df-4ea9-8639-a7a7cb18e577"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:48.016020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.015997 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-data" pod "bf6842a8-36df-4ea9-8639-a7a7cb18e577" (UID: "bf6842a8-36df-4ea9-8639-a7a7cb18e577"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:48.016186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.016164 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-certs" pod "bf6842a8-36df-4ea9-8639-a7a7cb18e577" (UID: "bf6842a8-36df-4ea9-8639-a7a7cb18e577"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:48.019367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.019338 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-envoy" pod "bf6842a8-36df-4ea9-8639-a7a7cb18e577" (UID: "bf6842a8-36df-4ea9-8639-a7a7cb18e577"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:48.019495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.019370 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-podinfo" pod "bf6842a8-36df-4ea9-8639-a7a7cb18e577" (UID: "bf6842a8-36df-4ea9-8639-a7a7cb18e577"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:48.019495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.019470 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-kube-api-access-ggnqp" pod "bf6842a8-36df-4ea9-8639-a7a7cb18e577" (UID: "bf6842a8-36df-4ea9-8639-a7a7cb18e577"). InnerVolumeSpecName "kube-api-access-ggnqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:48.021132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.021106 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-token" pod "bf6842a8-36df-4ea9-8639-a7a7cb18e577" (UID: "bf6842a8-36df-4ea9-8639-a7a7cb18e577"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:48.115609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.115522 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.115609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.115562 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggnqp\" (UniqueName: \"kubernetes.io/projected/bf6842a8-36df-4ea9-8639-a7a7cb18e577-kube-api-access-ggnqp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.115609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.115580 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.115609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.115595 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.115609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.115608 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.116000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.115623 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.116000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.115636 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.116000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.115649 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6842a8-36df-4ea9-8639-a7a7cb18e577-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.116000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.115661 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6842a8-36df-4ea9-8639-a7a7cb18e577-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.123819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.123785 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bcadd15-1201-491a-8348-dd39b94f9bae" path="/var/lib/kubelet/pods/1bcadd15-1201-491a-8348-dd39b94f9bae/volumes" Oct 30 06:17:48.587080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.586991 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:17:48.587080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.587061 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:17:48.739757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.739702 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" event={"ID":"bf6842a8-36df-4ea9-8639-a7a7cb18e577","Type":"ContainerDied","Data":"df21455ca848e33cdf150b81bbca79f01169d1863457f6b5474544e7328277c8"} Oct 30 06:17:48.739958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.739728 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2" Oct 30 06:17:48.739958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.739797 5001 scope.go:122] "RemoveContainer" containerID="6e62ce5bdcd3faefa71532b83323531c5d0d9251562db1a2df7cf641e4e919ae" Oct 30 06:17:48.781337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.777597 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:17:48.786589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.786516 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:17:48.823118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.823059 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2"] Oct 30 06:17:48.836374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:48.836340 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f7a606-openshift-default-6887bbc9cf-wc2s2"] Oct 30 06:17:49.589544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.589120 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:17:49.589544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.589190 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:17:49.601014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.600978 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd"] Oct 30 06:17:49.601662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601646 5001 cpu_manager.go:414] "containerMap: removing container" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerName="istio-proxy" Oct 30 06:17:49.601761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601664 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerName="istio-proxy" Oct 30 06:17:49.601761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601700 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" Oct 30 06:17:49.601761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601705 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" Oct 30 06:17:49.601761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601717 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" Oct 30 06:17:49.601761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601722 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" Oct 30 06:17:49.601761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601731 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ad5ce104-9403-4712-8512-9845efc5ac51" containerName="istio-proxy" Oct 30 06:17:49.601761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601735 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5ce104-9403-4712-8512-9845efc5ac51" containerName="istio-proxy" Oct 30 06:17:49.601761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601742 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1bcadd15-1201-491a-8348-dd39b94f9bae" containerName="istio-proxy" Oct 30 06:17:49.601761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601746 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bcadd15-1201-491a-8348-dd39b94f9bae" containerName="istio-proxy" Oct 30 06:17:49.602189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601851 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ed7d68-efcb-424b-8289-ad2c278768c2" containerName="istio-proxy" Oct 30 06:17:49.602189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601866 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" Oct 30 06:17:49.602189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601873 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" containerName="istio-proxy" Oct 30 06:17:49.602189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601880 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5ce104-9403-4712-8512-9845efc5ac51" containerName="istio-proxy" Oct 30 06:17:49.602189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.601884 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bcadd15-1201-491a-8348-dd39b94f9bae" containerName="istio-proxy" Oct 30 06:17:49.642024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.641996 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.722918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.722883 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd"] Oct 30 06:17:49.804761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.804722 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.804979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.804798 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.804979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.804849 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.804979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.804898 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.804979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.804922 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.805188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.805071 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.805188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.805113 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg48b\" (UniqueName: \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-kube-api-access-mg48b\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.805188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.805139 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.805188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.805156 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.906085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.906048 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.906425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.906392 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg48b\" (UniqueName: \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-kube-api-access-mg48b\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.906588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.906454 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.906588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.906482 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.906588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.906510 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.906588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.906529 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.906796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.906611 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.906796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.906661 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.906796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.906716 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.906796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.906755 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.906997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.906798 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.907147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.907123 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.907214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.907177 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.907932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.907894 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.909235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.909208 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.909508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.909484 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.932561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.932524 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg48b\" (UniqueName: \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-kube-api-access-mg48b\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:49.933400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:49.933376 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:50.075968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.075927 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:50.119590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.119554 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6842a8-36df-4ea9-8639-a7a7cb18e577" path="/var/lib/kubelet/pods/bf6842a8-36df-4ea9-8639-a7a7cb18e577/volumes" Oct 30 06:17:50.455344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.455291 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd"] Oct 30 06:17:50.456134 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:17:50.456105 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382c841d_ed3a_43fe_bd5f_76a1ba0aaeb6.slice/crio-45500014d991344b1bea96e5524b8784032ebeaf3809a665bae41537e7db2cb8 WatchSource:0}: Error finding container 45500014d991344b1bea96e5524b8784032ebeaf3809a665bae41537e7db2cb8: Status 404 returned error can't find the container with id 45500014d991344b1bea96e5524b8784032ebeaf3809a665bae41537e7db2cb8 Oct 30 06:17:50.459184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.459142 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:50.459288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.459232 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:50.459288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.459275 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:50.614093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.591571 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:17:50.614093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.593872 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:17:50.794418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.793432 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" event={"ID":"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6","Type":"ContainerStarted","Data":"3bdc572672d7c68e842a3ddbde11b7ff56200c0746529025e3797669be018705"} Oct 30 06:17:50.794418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.793499 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" event={"ID":"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6","Type":"ContainerStarted","Data":"45500014d991344b1bea96e5524b8784032ebeaf3809a665bae41537e7db2cb8"} Oct 30 06:17:50.796832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.796804 5001 generic.go:356] "Generic (PLEG): container finished" podID="21141de6-8bbc-4245-9c58-d4218a32169c" containerID="4523f586adf5e1f1d1d98c5cb05be0d3bda229178280f153e194ae473e2b2959" exitCode=0 Oct 30 06:17:50.798048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.797032 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" event={"ID":"21141de6-8bbc-4245-9c58-d4218a32169c","Type":"ContainerDied","Data":"4523f586adf5e1f1d1d98c5cb05be0d3bda229178280f153e194ae473e2b2959"} Oct 30 06:17:50.798048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.797553 5001 scope.go:122] "RemoveContainer" containerID="4523f586adf5e1f1d1d98c5cb05be0d3bda229178280f153e194ae473e2b2959" Oct 30 06:17:50.806263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.800624 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:50.806720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.806657 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:50.806923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.806902 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:50.847557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:50.847508 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" podStartSLOduration=1.847493644 podStartE2EDuration="1.847493644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:50.843177971 +0000 UTC m=+1484.839134980" watchObservedRunningTime="2026-10-30 06:17:50.847493644 +0000 UTC m=+1484.843450654" Oct 30 06:17:51.076158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.076067 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:51.077768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.077737 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 06:17:51.077903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.077800 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 06:17:51.199120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.199077 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f"] Oct 30 06:17:51.214585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.214140 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:17:51.217016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.216981 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f"] Oct 30 06:17:51.251605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.251570 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j"] Oct 30 06:17:51.263018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.262255 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-532f3b-dockercfg-9bpnj\"" Oct 30 06:17:51.263018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.262348 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-532f3b\"" Oct 30 06:17:51.304385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.301726 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:17:51.345381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.344753 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh26p\" (UniqueName: \"kubernetes.io/projected/a5bda487-9e0e-40da-8340-7577ab18ca42-kube-api-access-gh26p\") pod \"payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f\" (UID: \"a5bda487-9e0e-40da-8340-7577ab18ca42\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:17:51.345381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.344803 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcce932-f11c-45af-b2a8-3563da5b90ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j\" (UID: \"ddcce932-f11c-45af-b2a8-3563da5b90ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:17:51.345381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.344933 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2wt\" (UniqueName: \"kubernetes.io/projected/ddcce932-f11c-45af-b2a8-3563da5b90ac-kube-api-access-6h2wt\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j\" (UID: \"ddcce932-f11c-45af-b2a8-3563da5b90ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:17:51.345381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.344965 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bda487-9e0e-40da-8340-7577ab18ca42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f\" (UID: \"a5bda487-9e0e-40da-8340-7577ab18ca42\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:17:51.433061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.433024 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j"] Oct 30 06:17:51.521650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.521616 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh26p\" (UniqueName: \"kubernetes.io/projected/a5bda487-9e0e-40da-8340-7577ab18ca42-kube-api-access-gh26p\") pod \"payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f\" (UID: \"a5bda487-9e0e-40da-8340-7577ab18ca42\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:17:51.543984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.543946 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcce932-f11c-45af-b2a8-3563da5b90ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j\" (UID: \"ddcce932-f11c-45af-b2a8-3563da5b90ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:17:51.592592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.592552 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2wt\" (UniqueName: \"kubernetes.io/projected/ddcce932-f11c-45af-b2a8-3563da5b90ac-kube-api-access-6h2wt\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j\" (UID: \"ddcce932-f11c-45af-b2a8-3563da5b90ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:17:51.601481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.601401 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bda487-9e0e-40da-8340-7577ab18ca42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f\" (UID: \"a5bda487-9e0e-40da-8340-7577ab18ca42\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:17:51.635605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.602570 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bda487-9e0e-40da-8340-7577ab18ca42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f\" (UID: \"a5bda487-9e0e-40da-8340-7577ab18ca42\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:17:51.648962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.637699 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcce932-f11c-45af-b2a8-3563da5b90ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j\" (UID: \"ddcce932-f11c-45af-b2a8-3563da5b90ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:17:51.679176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.678700 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh26p\" (UniqueName: \"kubernetes.io/projected/a5bda487-9e0e-40da-8340-7577ab18ca42-kube-api-access-gh26p\") pod \"payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f\" (UID: \"a5bda487-9e0e-40da-8340-7577ab18ca42\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:17:51.710747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.710705 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2wt\" (UniqueName: \"kubernetes.io/projected/ddcce932-f11c-45af-b2a8-3563da5b90ac-kube-api-access-6h2wt\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j\" (UID: \"ddcce932-f11c-45af-b2a8-3563da5b90ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:17:51.823811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.823778 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:17:51.831658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.831626 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" event={"ID":"21141de6-8bbc-4245-9c58-d4218a32169c","Type":"ContainerStarted","Data":"28704140959d6feed6149018ed61727fccd0d96faf5198fd0011cc1cdb2e09d8"} Oct 30 06:17:51.831808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:51.831733 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:17:52.033490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.031289 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:17:52.033490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.032054 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:17:52.033490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.032100 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:17:52.100341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.093410 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:17:52.120610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.113450 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 06:17:52.120610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.113513 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 06:17:52.312632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.312522 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j"] Oct 30 06:17:52.324205 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:17:52.324166 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcce932_f11c_45af_b2a8_3563da5b90ac.slice/crio-54a5cc137323b3e5aa76a4dedb36ff23a1242bc01be7b6dfc8ced8204b79c71b WatchSource:0}: Error finding container 54a5cc137323b3e5aa76a4dedb36ff23a1242bc01be7b6dfc8ced8204b79c71b: Status 404 returned error can't find the container with id 54a5cc137323b3e5aa76a4dedb36ff23a1242bc01be7b6dfc8ced8204b79c71b Oct 30 06:17:52.515205 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:17:52.514960 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bda487_9e0e_40da_8340_7577ab18ca42.slice/crio-3646c635a906878222c25c0089eef1090ac0d69e575015f0bd52d808ebf17a8e WatchSource:0}: Error finding container 3646c635a906878222c25c0089eef1090ac0d69e575015f0bd52d808ebf17a8e: Status 404 returned error can't find the container with id 3646c635a906878222c25c0089eef1090ac0d69e575015f0bd52d808ebf17a8e Oct 30 06:17:52.523398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.523363 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f"] Oct 30 06:17:52.841344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.841235 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" event={"ID":"ddcce932-f11c-45af-b2a8-3563da5b90ac","Type":"ContainerStarted","Data":"1aaf702316d6eeb1e1e71ce52a9aad979f368cec5d4e317f6376cb36f1cf9431"} Oct 30 06:17:52.841344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.841300 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" event={"ID":"ddcce932-f11c-45af-b2a8-3563da5b90ac","Type":"ContainerStarted","Data":"54a5cc137323b3e5aa76a4dedb36ff23a1242bc01be7b6dfc8ced8204b79c71b"} Oct 30 06:17:52.841778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.841391 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:17:52.844168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.844144 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" event={"ID":"a5bda487-9e0e-40da-8340-7577ab18ca42","Type":"ContainerStarted","Data":"e34e55f7b2f6c1337d66312e91ee86e3cf66bd32157fe12ff95bb1d8a8e6c947"} Oct 30 06:17:52.844288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.844182 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:17:52.844288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.844198 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" event={"ID":"a5bda487-9e0e-40da-8340-7577ab18ca42","Type":"ContainerStarted","Data":"3646c635a906878222c25c0089eef1090ac0d69e575015f0bd52d808ebf17a8e"} Oct 30 06:17:52.905332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.905022 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" podStartSLOduration=1.9049994529999998 podStartE2EDuration="1.904999453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:52.898705092 +0000 UTC m=+1486.894662104" watchObservedRunningTime="2026-10-30 06:17:52.904999453 +0000 UTC m=+1486.900956464" Oct 30 06:17:52.961822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:52.961755 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" podStartSLOduration=1.961731144 podStartE2EDuration="1.961731144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:52.952618609 +0000 UTC m=+1486.948575620" watchObservedRunningTime="2026-10-30 06:17:52.961731144 +0000 UTC m=+1486.957688155" Oct 30 06:17:53.025482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:53.025427 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:17:53.025659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:53.025509 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:17:53.185351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:53.162021 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 06:17:53.185816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:53.185770 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 06:17:53.970333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:53.969708 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:17:53.970333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:53.969767 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:17:54.081164 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:54.081126 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 06:17:54.081469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:54.081438 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 06:17:54.970329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:54.969773 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:17:54.970329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:54.969835 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:17:55.079603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:55.079565 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 06:17:55.079869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:55.079839 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 06:17:55.779749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:55.778796 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:17:55.970764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:55.970727 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:17:55.971392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:55.971358 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:17:56.078905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:56.078816 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 06:17:56.079610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:56.079151 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 06:17:56.986335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:56.972079 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:17:56.986335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:56.972141 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:17:57.139817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:57.139778 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 06:17:57.140103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:57.140072 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 06:17:57.971902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:57.971733 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:17:57.971902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:57.971796 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:17:58.123239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:58.123198 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:58.123754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:58.123718 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:58.124836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:58.124810 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:17:58.743636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:58.743602 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:17:58.982634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:58.982597 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:17:58.986917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:58.986881 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:17:59.241781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:17:59.241431 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j"] Oct 30 06:18:00.541380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.529813 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b"] Oct 30 06:18:00.750336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.724379 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b"] Oct 30 06:18:00.750336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.728095 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.798794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.793419 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a55a4b-openshift-default-dockercfg-6brvj\"" Oct 30 06:18:00.875252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.875207 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-data\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.875654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.875628 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-socket\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.875884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.875865 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-credential-socket\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.876096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.876079 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istiod-ca-cert\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.876344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.876326 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-podinfo\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.876549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.876531 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-certs\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.884333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.876728 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-envoy\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.884640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.884614 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-token\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.884831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.884812 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4hxb\" (UniqueName: \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-kube-api-access-v4hxb\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.986970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.986933 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-certs\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.987187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.987036 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-envoy\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.987440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.987412 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-certs\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.987563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.987544 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-token\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.987624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.987611 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4hxb\" (UniqueName: \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-kube-api-access-v4hxb\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.987687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.987657 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-data\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.987742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.987717 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-socket\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.987808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.987788 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-credential-socket\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.987878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.987863 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istiod-ca-cert\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.987997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.987980 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-podinfo\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.988177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.988156 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-data\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.988747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.988720 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-socket\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.989085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.989059 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istiod-ca-cert\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.991367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.989371 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-credential-socket\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.991367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.990685 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-podinfo\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:00.993477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:00.993456 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-envoy\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:01.011583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:01.011552 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-token\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:01.011729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:01.011653 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4hxb\" (UniqueName: \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-kube-api-access-v4hxb\") pod \"e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:01.120628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:01.120582 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:01.491627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:01.491594 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b"] Oct 30 06:18:01.492182 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:18:01.492152 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82997d1a_74de_4f2f_b8cb_47aa1fab159c.slice/crio-88c6f3d79f2dff3c48caadc895a1b9013678e37a1b670521ccd0573c895d2889 WatchSource:0}: Error finding container 88c6f3d79f2dff3c48caadc895a1b9013678e37a1b670521ccd0573c895d2889: Status 404 returned error can't find the container with id 88c6f3d79f2dff3c48caadc895a1b9013678e37a1b670521ccd0573c895d2889 Oct 30 06:18:01.495498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:01.495457 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:01.495586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:01.495544 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:01.495622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:01.495585 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:02.014147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.014109 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" event={"ID":"82997d1a-74de-4f2f-b8cb-47aa1fab159c","Type":"ContainerStarted","Data":"4a2e64bafe5b33136118b72d8e7eff8112424a287d34e962d7dc85a13908b56b"} Oct 30 06:18:02.014673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.014171 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" event={"ID":"82997d1a-74de-4f2f-b8cb-47aa1fab159c","Type":"ContainerStarted","Data":"88c6f3d79f2dff3c48caadc895a1b9013678e37a1b670521ccd0573c895d2889"} Oct 30 06:18:02.094587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.094537 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" podStartSLOduration=2.094519196 podStartE2EDuration="2.094519196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:02.07070794 +0000 UTC m=+1496.066664951" watchObservedRunningTime="2026-10-30 06:18:02.094519196 +0000 UTC m=+1496.090476206" Oct 30 06:18:02.120852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.120750 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:02.122607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.122582 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:18:02.122730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.122627 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 06:18:02.139484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.139450 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j"] Oct 30 06:18:02.144458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.144435 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.193343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.179679 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8272af-openshift-default-dockercfg-x7sv9\"" Oct 30 06:18:02.200070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.200027 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j"] Oct 30 06:18:02.216987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.216601 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-podinfo\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.216987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.216636 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-data\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.216987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.216777 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-credential-socket\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.216987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.216805 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-token\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.216987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.216828 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-certs\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.216987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.216849 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-socket\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.216987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.216891 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-envoy\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.216987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.216917 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istiod-ca-cert\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.216987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.216942 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5gj\" (UniqueName: \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-kube-api-access-bp5gj\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.322573 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-envoy\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.322644 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istiod-ca-cert\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.322688 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5gj\" (UniqueName: \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-kube-api-access-bp5gj\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.322764 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-podinfo\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.322796 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-data\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.322949 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-credential-socket\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.322988 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-token\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.323027 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-certs\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.323062 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-socket\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.323597 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-socket\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.324235 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-data\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.333399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.324877 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istiod-ca-cert\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.335003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.334973 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-credential-socket\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.347698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.335452 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-certs\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.347951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.335634 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-envoy\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.414022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.399140 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-token\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.414022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.402133 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5gj\" (UniqueName: \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-kube-api-access-bp5gj\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.414022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.402968 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-podinfo\") pod \"e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:02.513332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:02.512778 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:03.125230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:03.125201 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:03.277861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:03.277828 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j"] Oct 30 06:18:03.282509 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:18:03.282471 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cdba52_fb35_42fa_91f2_77ecec35b5ef.slice/crio-e9c3730d92cdd9ad9f3ac841c80be39201c7135b1a1ef22c1e1d0d265c0a5154 WatchSource:0}: Error finding container e9c3730d92cdd9ad9f3ac841c80be39201c7135b1a1ef22c1e1d0d265c0a5154: Status 404 returned error can't find the container with id e9c3730d92cdd9ad9f3ac841c80be39201c7135b1a1ef22c1e1d0d265c0a5154 Oct 30 06:18:03.286110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:03.286062 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:03.286245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:03.286161 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:03.286245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:03.286207 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:03.855996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:03.855962 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:18:03.878323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:03.878281 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:18:04.092545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.092501 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" event={"ID":"02cdba52-fb35-42fa-91f2-77ecec35b5ef","Type":"ContainerStarted","Data":"e5f44e7b4363c1a0ad75843d9feb6b8c070f25f0212576bfa3681c2b90f5100f"} Oct 30 06:18:04.092756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.092605 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" event={"ID":"02cdba52-fb35-42fa-91f2-77ecec35b5ef","Type":"ContainerStarted","Data":"e9c3730d92cdd9ad9f3ac841c80be39201c7135b1a1ef22c1e1d0d265c0a5154"} Oct 30 06:18:04.093025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.093001 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:04.094103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.094063 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:18:04.142593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.142490 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" podStartSLOduration=2.142474949 podStartE2EDuration="2.142474949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:04.140711291 +0000 UTC m=+1498.136668302" watchObservedRunningTime="2026-10-30 06:18:04.142474949 +0000 UTC m=+1498.138431960" Oct 30 06:18:04.242540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.242496 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" podUID="9a558ec8-ad54-423a-a67b-75b7dc613bdc" containerName="payload-processing" containerID="cri-o://a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972" gracePeriod=25 Oct 30 06:18:04.514055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.513975 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:04.524253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.523893 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:04.593565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.593541 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:18:04.686383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.686228 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a558ec8-ad54-423a-a67b-75b7dc613bdc-kube-api-access-nbcm9\" (UniqueName: \"kubernetes.io/projected/9a558ec8-ad54-423a-a67b-75b7dc613bdc-kube-api-access-nbcm9\") pod \"9a558ec8-ad54-423a-a67b-75b7dc613bdc\" (UID: \"9a558ec8-ad54-423a-a67b-75b7dc613bdc\") " Oct 30 06:18:04.686579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.686403 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a558ec8-ad54-423a-a67b-75b7dc613bdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a558ec8-ad54-423a-a67b-75b7dc613bdc-plugins-config-volume\") pod \"9a558ec8-ad54-423a-a67b-75b7dc613bdc\" (UID: \"9a558ec8-ad54-423a-a67b-75b7dc613bdc\") " Oct 30 06:18:04.689373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.686842 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a558ec8-ad54-423a-a67b-75b7dc613bdc-plugins-config-volume" pod "9a558ec8-ad54-423a-a67b-75b7dc613bdc" (UID: "9a558ec8-ad54-423a-a67b-75b7dc613bdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:04.689373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.687523 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a558ec8-ad54-423a-a67b-75b7dc613bdc-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:04.698411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.698353 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a558ec8-ad54-423a-a67b-75b7dc613bdc-kube-api-access-nbcm9" pod "9a558ec8-ad54-423a-a67b-75b7dc613bdc" (UID: "9a558ec8-ad54-423a-a67b-75b7dc613bdc"). InnerVolumeSpecName "kube-api-access-nbcm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:04.789515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:04.789475 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbcm9\" (UniqueName: \"kubernetes.io/projected/9a558ec8-ad54-423a-a67b-75b7dc613bdc-kube-api-access-nbcm9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:05.046271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.046179 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp"] Oct 30 06:18:05.056105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.056068 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-501ef623-hdrjp"] Oct 30 06:18:05.104021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.103982 5001 generic.go:356] "Generic (PLEG): container finished" podID="9a558ec8-ad54-423a-a67b-75b7dc613bdc" containerID="a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972" exitCode=0 Oct 30 06:18:05.104220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.104029 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" Oct 30 06:18:05.104220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.104065 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" event={"ID":"9a558ec8-ad54-423a-a67b-75b7dc613bdc","Type":"ContainerDied","Data":"a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972"} Oct 30 06:18:05.104220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.104125 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j" event={"ID":"9a558ec8-ad54-423a-a67b-75b7dc613bdc","Type":"ContainerDied","Data":"5c6c1df16e9ba07f106c4b1b29352a4d4d9bd78de76009e00349aeab58e48c05"} Oct 30 06:18:05.104220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.104152 5001 scope.go:122] "RemoveContainer" containerID="a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972" Oct 30 06:18:05.105627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.105414 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:05.106426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.106405 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:18:05.117547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.117521 5001 scope.go:122] "RemoveContainer" containerID="a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972" Oct 30 06:18:05.117903 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:18:05.117883 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972\": container with ID starting with a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972 not found: ID does not exist" containerID="a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972" Oct 30 06:18:05.117985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.117915 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972"} err="failed to get container status \"a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972\": rpc error: code = NotFound desc = could not find container \"a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972\": container with ID starting with a203c94522570822a7149d01187d57fd2f230527bd88e3cd99a27248507de972 not found: ID does not exist" Oct 30 06:18:05.210146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.210112 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j"] Oct 30 06:18:05.220927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:05.220895 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-9d9bd6c84-5t78j"] Oct 30 06:18:06.122092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.122064 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a558ec8-ad54-423a-a67b-75b7dc613bdc" path="/var/lib/kubelet/pods/9a558ec8-ad54-423a-a67b-75b7dc613bdc/volumes" Oct 30 06:18:06.122526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.122512 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59" path="/var/lib/kubelet/pods/cb750e5f-87e7-4a1b-a9f6-5da3ccaaad59/volumes" Oct 30 06:18:06.403013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.402934 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:18:06.424826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.424799 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:18:06.641102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.641067 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm"] Oct 30 06:18:06.641818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.641802 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9a558ec8-ad54-423a-a67b-75b7dc613bdc" containerName="payload-processing" Oct 30 06:18:06.641883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.641820 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a558ec8-ad54-423a-a67b-75b7dc613bdc" containerName="payload-processing" Oct 30 06:18:06.641943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.641933 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a558ec8-ad54-423a-a67b-75b7dc613bdc" containerName="payload-processing" Oct 30 06:18:06.648016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.647960 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:18:06.667185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.667154 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm"] Oct 30 06:18:06.673225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.673200 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a55a4b\"" Oct 30 06:18:06.673986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.673477 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a55a4b-dockercfg-7x24p\"" Oct 30 06:18:06.741330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.734174 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsktp\" (UniqueName: \"kubernetes.io/projected/dae76461-bda0-4a23-bcf6-e74c3a4ee951-kube-api-access-nsktp\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm\" (UID: \"dae76461-bda0-4a23-bcf6-e74c3a4ee951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:18:06.741330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.734419 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae76461-bda0-4a23-bcf6-e74c3a4ee951-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm\" (UID: \"dae76461-bda0-4a23-bcf6-e74c3a4ee951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:18:06.833385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.828866 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx"] Oct 30 06:18:06.837288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.837250 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsktp\" (UniqueName: \"kubernetes.io/projected/dae76461-bda0-4a23-bcf6-e74c3a4ee951-kube-api-access-nsktp\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm\" (UID: \"dae76461-bda0-4a23-bcf6-e74c3a4ee951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:18:06.837629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.837608 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae76461-bda0-4a23-bcf6-e74c3a4ee951-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm\" (UID: \"dae76461-bda0-4a23-bcf6-e74c3a4ee951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:18:06.838704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.838680 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae76461-bda0-4a23-bcf6-e74c3a4ee951-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm\" (UID: \"dae76461-bda0-4a23-bcf6-e74c3a4ee951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:18:06.839377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:06.839359 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:18:07.027452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.026353 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsktp\" (UniqueName: \"kubernetes.io/projected/dae76461-bda0-4a23-bcf6-e74c3a4ee951-kube-api-access-nsktp\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm\" (UID: \"dae76461-bda0-4a23-bcf6-e74c3a4ee951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:18:07.060532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.060487 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx"] Oct 30 06:18:07.106422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.106384 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ebf91c-2596-4e19-93cb-566052ea639a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx\" (UID: \"86ebf91c-2596-4e19-93cb-566052ea639a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:18:07.106635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.106438 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dffwc\" (UniqueName: \"kubernetes.io/projected/86ebf91c-2596-4e19-93cb-566052ea639a-kube-api-access-dffwc\") pod \"payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx\" (UID: \"86ebf91c-2596-4e19-93cb-566052ea639a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:18:07.208817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.208782 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ebf91c-2596-4e19-93cb-566052ea639a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx\" (UID: \"86ebf91c-2596-4e19-93cb-566052ea639a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:18:07.209139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.209112 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dffwc\" (UniqueName: \"kubernetes.io/projected/86ebf91c-2596-4e19-93cb-566052ea639a-kube-api-access-dffwc\") pod \"payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx\" (UID: \"86ebf91c-2596-4e19-93cb-566052ea639a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:18:07.209615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.209557 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ebf91c-2596-4e19-93cb-566052ea639a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx\" (UID: \"86ebf91c-2596-4e19-93cb-566052ea639a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:18:07.233663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.233626 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dffwc\" (UniqueName: \"kubernetes.io/projected/86ebf91c-2596-4e19-93cb-566052ea639a-kube-api-access-dffwc\") pod \"payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx\" (UID: \"86ebf91c-2596-4e19-93cb-566052ea639a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:18:07.266810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.266773 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:18:07.344876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.343197 5001 scope.go:122] "RemoveContainer" containerID="8a964246a75863c36a41ba02e7a4524b17fa4cd727d70247f7949a2be11c0914" Oct 30 06:18:07.365891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.365786 5001 scope.go:122] "RemoveContainer" containerID="8b1187ff20cde8189a2cb11693387532fdef1adb84380c172b288a6c7da6cddc" Oct 30 06:18:07.390208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.390044 5001 scope.go:122] "RemoveContainer" containerID="c3662750d72d61bce7a02c023b918e8e1715e6a4f0a652530f654b72ee63c3ac" Oct 30 06:18:07.410502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.410470 5001 scope.go:122] "RemoveContainer" containerID="c58d049958aaa3532df5b8aa5a1a34547c31b94355383c22f4360373ce2071a3" Oct 30 06:18:07.442808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.442785 5001 scope.go:122] "RemoveContainer" containerID="759242ef7088cbaf57fa9f2d346025d0aa2567831616b54bf9a1465a33b45d6f" Oct 30 06:18:07.453398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.453379 5001 scope.go:122] "RemoveContainer" containerID="045fde5882ad25385ee17b2f9e09e4fc34e2410ac4f6298c3e0058756ca64a47" Oct 30 06:18:07.466506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.466484 5001 scope.go:122] "RemoveContainer" containerID="d0dbbccfb4e0a835d48eb66e5bda5c79632570dc7eedb19e60468e7991c4dc02" Oct 30 06:18:07.524241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.524214 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:18:07.596417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.596387 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm"] Oct 30 06:18:07.611396 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:18:07.611171 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae76461_bda0_4a23_bcf6_e74c3a4ee951.slice/crio-95e4bf4a9a50c8623b9c82f859e9dc7ff7962dcced44be1e72590dc69b27d22b WatchSource:0}: Error finding container 95e4bf4a9a50c8623b9c82f859e9dc7ff7962dcced44be1e72590dc69b27d22b: Status 404 returned error can't find the container with id 95e4bf4a9a50c8623b9c82f859e9dc7ff7962dcced44be1e72590dc69b27d22b Oct 30 06:18:07.866223 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:18:07.866188 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ebf91c_2596_4e19_93cb_566052ea639a.slice/crio-eace102f7af62019ed2eb4c3875afd4a9001f45992266f1d81237a9adab71e0e WatchSource:0}: Error finding container eace102f7af62019ed2eb4c3875afd4a9001f45992266f1d81237a9adab71e0e: Status 404 returned error can't find the container with id eace102f7af62019ed2eb4c3875afd4a9001f45992266f1d81237a9adab71e0e Oct 30 06:18:07.866588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:07.866561 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx"] Oct 30 06:18:08.136450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.136344 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" event={"ID":"dae76461-bda0-4a23-bcf6-e74c3a4ee951","Type":"ContainerStarted","Data":"f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d"} Oct 30 06:18:08.136450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.136405 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" event={"ID":"dae76461-bda0-4a23-bcf6-e74c3a4ee951","Type":"ContainerStarted","Data":"95e4bf4a9a50c8623b9c82f859e9dc7ff7962dcced44be1e72590dc69b27d22b"} Oct 30 06:18:08.136663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.136452 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:18:08.138187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.138152 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" event={"ID":"86ebf91c-2596-4e19-93cb-566052ea639a","Type":"ContainerStarted","Data":"9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e"} Oct 30 06:18:08.138350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.138205 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" event={"ID":"86ebf91c-2596-4e19-93cb-566052ea639a","Type":"ContainerStarted","Data":"eace102f7af62019ed2eb4c3875afd4a9001f45992266f1d81237a9adab71e0e"} Oct 30 06:18:08.138350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.138236 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:18:08.140024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.140000 5001 generic.go:356] "Generic (PLEG): container finished" podID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerID="8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa" exitCode=0 Oct 30 06:18:08.140117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.140067 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" event={"ID":"f072d364-6b3a-4b8b-9ae8-b9ff16503cda","Type":"ContainerDied","Data":"8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa"} Oct 30 06:18:08.140489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.140473 5001 scope.go:122] "RemoveContainer" containerID="8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa" Oct 30 06:18:08.141991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.141963 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:08.142063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.142034 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:08.142102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.142067 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:08.183821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:08.183776 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" podStartSLOduration=2.183763594 podStartE2EDuration="2.183763594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:08.180448283 +0000 UTC m=+1502.176405290" watchObservedRunningTime="2026-10-30 06:18:08.183763594 +0000 UTC m=+1502.179720604" Oct 30 06:18:08.291400 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:18:08.287879 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:18:09.212339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:09.210224 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" event={"ID":"f072d364-6b3a-4b8b-9ae8-b9ff16503cda","Type":"ContainerStarted","Data":"110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755"} Oct 30 06:18:09.294625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:09.294567 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" podStartSLOduration=3.294551719 podStartE2EDuration="3.294551719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:08.287738358 +0000 UTC m=+1502.283695369" watchObservedRunningTime="2026-10-30 06:18:09.294551719 +0000 UTC m=+1503.290508729" Oct 30 06:18:09.850091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:09.850052 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:18:09.851243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:09.851215 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 06:18:09.851412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:09.851263 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 06:18:10.849693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:10.849660 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 06:18:10.850077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:10.849712 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 06:18:11.849679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:11.849647 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 06:18:11.849960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:11.849699 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 06:18:12.709547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:12.709489 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:18:12.849920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:12.849882 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 06:18:12.850099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:12.849945 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 06:18:13.849445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:13.849415 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 06:18:13.849621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:13.849466 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 06:18:14.854265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:14.854231 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:18:14.855628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:14.855603 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:18:15.068977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:15.068941 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc"] Oct 30 06:18:15.075780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:15.075751 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d086f215-gbvqc"] Oct 30 06:18:16.126631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:16.126592 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d553e23-eecd-4158-ad64-6470e751d604" path="/var/lib/kubelet/pods/8d553e23-eecd-4158-ad64-6470e751d604/volumes" Oct 30 06:18:19.213140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:19.213115 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:18:19.213626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:19.213176 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:18:19.785585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:19.785528 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s"] Oct 30 06:18:19.918039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:19.917991 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:18:19.923663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:19.923616 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s"] Oct 30 06:18:19.999353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:19.999280 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f50f6e\"" Oct 30 06:18:19.999802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:19.999748 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f50f6e-dockercfg-gw6fl\"" Oct 30 06:18:20.076410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:20.066393 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4mr\" (UniqueName: \"kubernetes.io/projected/f21ab9cb-4b51-467e-83d9-dbb0f9386660-kube-api-access-wv4mr\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s\" (UID: \"f21ab9cb-4b51-467e-83d9-dbb0f9386660\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:18:20.076410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:20.066659 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21ab9cb-4b51-467e-83d9-dbb0f9386660-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s\" (UID: \"f21ab9cb-4b51-467e-83d9-dbb0f9386660\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:18:20.168604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:20.168560 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4mr\" (UniqueName: \"kubernetes.io/projected/f21ab9cb-4b51-467e-83d9-dbb0f9386660-kube-api-access-wv4mr\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s\" (UID: \"f21ab9cb-4b51-467e-83d9-dbb0f9386660\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:18:20.168886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:20.168784 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21ab9cb-4b51-467e-83d9-dbb0f9386660-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s\" (UID: \"f21ab9cb-4b51-467e-83d9-dbb0f9386660\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:18:20.176353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:20.169819 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21ab9cb-4b51-467e-83d9-dbb0f9386660-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s\" (UID: \"f21ab9cb-4b51-467e-83d9-dbb0f9386660\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:18:20.287732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:20.287699 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4mr\" (UniqueName: \"kubernetes.io/projected/f21ab9cb-4b51-467e-83d9-dbb0f9386660-kube-api-access-wv4mr\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s\" (UID: \"f21ab9cb-4b51-467e-83d9-dbb0f9386660\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:18:20.309621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:20.309584 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:18:20.696331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:20.696266 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s"] Oct 30 06:18:20.703535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:20.703092 5001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:18:21.384893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:21.384819 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" event={"ID":"f21ab9cb-4b51-467e-83d9-dbb0f9386660","Type":"ContainerStarted","Data":"51c2163792ab89a07969a4ed6ebbc481868b4ec412d0f57f5946d05bc5709832"} Oct 30 06:18:24.432264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:24.432227 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" event={"ID":"f21ab9cb-4b51-467e-83d9-dbb0f9386660","Type":"ContainerStarted","Data":"4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f"} Oct 30 06:18:24.432727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:24.432324 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:18:28.060335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:28.058235 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" podStartSLOduration=6.410808188 podStartE2EDuration="9.058212299s" totalImagesPullingTime="2.647404111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:24.48479995 +0000 UTC m=+1518.480756984" watchObservedRunningTime="2026-10-30 06:18:28.058212299 +0000 UTC m=+1522.054169311" Oct 30 06:18:28.060931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:28.060904 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65"] Oct 30 06:18:28.102335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:28.098347 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4f2d58-wdt65"] Oct 30 06:18:28.154566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:28.153194 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1cb43cc-5dda-4675-9f59-29a42f2cdbd3" path="/var/lib/kubelet/pods/d1cb43cc-5dda-4675-9f59-29a42f2cdbd3/volumes" Oct 30 06:18:32.593509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:32.593455 5001 generic.go:356] "Generic (PLEG): container finished" podID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerID="18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30" exitCode=0 Oct 30 06:18:32.594092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:32.593516 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" event={"ID":"286fb2e3-76ff-44f6-aaf4-c537b0673ce5","Type":"ContainerDied","Data":"18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30"} Oct 30 06:18:32.594741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:32.594718 5001 scope.go:122] "RemoveContainer" containerID="18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30" Oct 30 06:18:32.596650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:32.596609 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:32.596723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:32.596705 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:32.596778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:32.596762 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:33.643332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:33.643036 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" event={"ID":"286fb2e3-76ff-44f6-aaf4-c537b0673ce5","Type":"ContainerStarted","Data":"9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21"} Oct 30 06:18:34.504559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.504528 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:18:34.504756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.504721 5001 patch_prober.go:29] interesting pod/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 06:18:34.504811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.504766 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 06:18:34.693504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.693461 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq"] Oct 30 06:18:34.697779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.697750 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:34.718781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.718753 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq"] Oct 30 06:18:34.726254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.726222 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:18:34.726467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.726299 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:18:34.726549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.726528 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:18:34.728126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.728111 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:18:34.739913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.739864 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb"] Oct 30 06:18:34.745565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.745539 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.800402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.800295 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.808482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.808450 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.808623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.808603 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jxt\" (UniqueName: \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-kube-api-access-s7jxt\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.808709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.808693 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.808940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.808919 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.809012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.808996 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxs2x\" (UniqueName: \"kubernetes.io/projected/91db3de6-a8d4-407f-86b7-80fb8076186c-kube-api-access-cxs2x\") pod \"maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:34.809069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.809041 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.809146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.809101 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.809238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.809219 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.809382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.809253 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.809382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.809284 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:34.809562 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.809539 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:34.813975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.813950 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb"] Oct 30 06:18:34.910706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.910673 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxs2x\" (UniqueName: \"kubernetes.io/projected/91db3de6-a8d4-407f-86b7-80fb8076186c-kube-api-access-cxs2x\") pod \"maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:34.910895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.910720 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.910895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.910745 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.910895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.910770 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.910895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.910785 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.910895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.910807 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:34.910895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.910829 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:34.910895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.910889 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.911396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.910934 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.911396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.911033 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jxt\" (UniqueName: \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-kube-api-access-s7jxt\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.911396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.911108 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.911396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.911167 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.911396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.911231 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.911396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.911247 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.911396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.911273 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.911818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.911480 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.911818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.911720 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:34.911818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.911753 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.913998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.913975 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:34.914194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.914173 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.914363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.914345 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.935491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.935466 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.935656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.935630 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jxt\" (UniqueName: \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-kube-api-access-s7jxt\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:34.935722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:34.935677 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxs2x\" (UniqueName: \"kubernetes.io/projected/91db3de6-a8d4-407f-86b7-80fb8076186c-kube-api-access-cxs2x\") pod \"maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:35.015231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.015194 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:35.119917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.119170 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:35.391294 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:18:35.391260 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91db3de6_a8d4_407f_86b7_80fb8076186c.slice/crio-769ed03ae5ec7b00d9034ab59f7513d29c37e5a671919d62780cf666cca4c91c WatchSource:0}: Error finding container 769ed03ae5ec7b00d9034ab59f7513d29c37e5a671919d62780cf666cca4c91c: Status 404 returned error can't find the container with id 769ed03ae5ec7b00d9034ab59f7513d29c37e5a671919d62780cf666cca4c91c Oct 30 06:18:35.392241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.392208 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq"] Oct 30 06:18:35.443254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.443228 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:18:35.493880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.493848 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb"] Oct 30 06:18:35.497084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.497047 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:35.497160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.497132 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:35.497209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.497179 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:35.505067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.505027 5001 patch_prober.go:29] interesting pod/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 06:18:35.505202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.505077 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 06:18:35.714513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.714482 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" event={"ID":"91db3de6-a8d4-407f-86b7-80fb8076186c","Type":"ContainerStarted","Data":"769ed03ae5ec7b00d9034ab59f7513d29c37e5a671919d62780cf666cca4c91c"} Oct 30 06:18:35.717668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.717645 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" event={"ID":"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83","Type":"ContainerStarted","Data":"6c4633cacaac77a9e4de1bb8f44c85e69307b7efc37b312db7c3fd0ff0304e7e"} Oct 30 06:18:35.717808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.717798 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" event={"ID":"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83","Type":"ContainerStarted","Data":"55171cdd5fc62d5ce89eaf039fdf9963f3f9ad7920f010c48c2066751eb7f92e"} Oct 30 06:18:35.779593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:35.779528 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" podStartSLOduration=1.779507047 podStartE2EDuration="1.779507047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:35.762876852 +0000 UTC m=+1529.758833891" watchObservedRunningTime="2026-10-30 06:18:35.779507047 +0000 UTC m=+1529.775464058" Oct 30 06:18:36.121956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:36.121917 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:18:36.122167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:36.121974 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:18:36.123550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:36.123525 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:36.287329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:36.286828 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:18:36.511668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:36.508077 5001 patch_prober.go:29] interesting pod/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 06:18:36.511668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:36.508135 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 06:18:36.737054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:36.737018 5001 generic.go:356] "Generic (PLEG): container finished" podID="91db3de6-a8d4-407f-86b7-80fb8076186c" containerID="0b2436c470a9ef9d3a545c1bfc2e9e062ff5b578da4cc710cc199450e656afb9" exitCode=0 Oct 30 06:18:36.737787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:36.737215 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" event={"ID":"91db3de6-a8d4-407f-86b7-80fb8076186c","Type":"ContainerDied","Data":"0b2436c470a9ef9d3a545c1bfc2e9e062ff5b578da4cc710cc199450e656afb9"} Oct 30 06:18:37.137801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.137768 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:18:37.138113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.138082 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:18:37.309363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.308546 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l"] Oct 30 06:18:37.317548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.317519 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.320644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.320618 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l"] Oct 30 06:18:37.332755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.332490 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-bc5d8f-openshift-default-dockercfg-7lqpp\"" Oct 30 06:18:37.455831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.445886 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-socket\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.455831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.446002 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63052f6-d65d-4307-93bb-daa7a5581e78-istiod-ca-cert\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.455831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.446162 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-credential-socket\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.455831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.446191 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-podinfo\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.455831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.446219 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-token\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.455831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.446502 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-certs\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.455831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.446540 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-envoy\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.455831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.446598 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqv8z\" (UniqueName: \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-kube-api-access-kqv8z\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.455831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.446691 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-data\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.525362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.521669 5001 patch_prober.go:29] interesting pod/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 06:18:37.525362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.521828 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.570053 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63052f6-d65d-4307-93bb-daa7a5581e78-istiod-ca-cert\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.570218 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-credential-socket\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.570247 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-podinfo\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.570276 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-token\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.570363 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-certs\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.570396 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-envoy\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.570435 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqv8z\" (UniqueName: \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-kube-api-access-kqv8z\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.570523 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-data\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.570566 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-socket\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.571060 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-socket\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.571685 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63052f6-d65d-4307-93bb-daa7a5581e78-istiod-ca-cert\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.584332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.571931 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-credential-socket\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.791389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.790239 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-podinfo\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.791892 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.791713 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-data\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.975333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.945343 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-certs\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.975333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.954848 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-token\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.975333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.964049 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqv8z\" (UniqueName: \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-kube-api-access-kqv8z\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:37.977231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:37.976461 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-envoy\") pod \"e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:38.124464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.123609 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:18:38.124464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.123960 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:18:38.247150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.247113 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:38.443983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.443946 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:38.532956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.532911 5001 patch_prober.go:29] interesting pod/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 06:18:38.533156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.533015 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 06:18:38.544964 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:18:38.544923 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:18:38.606334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.560294 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91db3de6-a8d4-407f-86b7-80fb8076186c-kube-api-access-cxs2x\" (UniqueName: \"kubernetes.io/projected/91db3de6-a8d4-407f-86b7-80fb8076186c-kube-api-access-cxs2x\") pod \"91db3de6-a8d4-407f-86b7-80fb8076186c\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " Oct 30 06:18:38.606334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.577695 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-xks-ca\") pod \"91db3de6-a8d4-407f-86b7-80fb8076186c\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " Oct 30 06:18:38.606334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.577866 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-openshift-ca\") pod \"91db3de6-a8d4-407f-86b7-80fb8076186c\" (UID: \"91db3de6-a8d4-407f-86b7-80fb8076186c\") " Oct 30 06:18:38.624364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.610831 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-xks-ca" pod "91db3de6-a8d4-407f-86b7-80fb8076186c" (UID: "91db3de6-a8d4-407f-86b7-80fb8076186c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:18:38.624364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.611194 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-openshift-ca" pod "91db3de6-a8d4-407f-86b7-80fb8076186c" (UID: "91db3de6-a8d4-407f-86b7-80fb8076186c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:38.631296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.631257 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91db3de6-a8d4-407f-86b7-80fb8076186c-kube-api-access-cxs2x" pod "91db3de6-a8d4-407f-86b7-80fb8076186c" (UID: "91db3de6-a8d4-407f-86b7-80fb8076186c"). InnerVolumeSpecName "kube-api-access-cxs2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:38.706806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.706771 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxs2x\" (UniqueName: \"kubernetes.io/projected/91db3de6-a8d4-407f-86b7-80fb8076186c-kube-api-access-cxs2x\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:38.706995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.706979 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:38.707098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.707085 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91db3de6-a8d4-407f-86b7-80fb8076186c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:38.934224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.934175 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l"] Oct 30 06:18:38.938288 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:18:38.938004 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63052f6_d65d_4307_93bb_daa7a5581e78.slice/crio-a77462158d1728d4a8424134765916b784bfa410f4f2223a30a2e6405ee6e956 WatchSource:0}: Error finding container a77462158d1728d4a8424134765916b784bfa410f4f2223a30a2e6405ee6e956: Status 404 returned error can't find the container with id a77462158d1728d4a8424134765916b784bfa410f4f2223a30a2e6405ee6e956 Oct 30 06:18:38.947346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.944891 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:38.947346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.945044 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:38.947346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:38.945133 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:39.143662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:39.125155 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:18:39.143662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:39.125231 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:18:39.217346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:39.215752 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" event={"ID":"a63052f6-d65d-4307-93bb-daa7a5581e78","Type":"ContainerStarted","Data":"a77462158d1728d4a8424134765916b784bfa410f4f2223a30a2e6405ee6e956"} Oct 30 06:18:39.319972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:39.319742 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" Oct 30 06:18:39.319972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:39.319747 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq" event={"ID":"91db3de6-a8d4-407f-86b7-80fb8076186c","Type":"ContainerDied","Data":"769ed03ae5ec7b00d9034ab59f7513d29c37e5a671919d62780cf666cca4c91c"} Oct 30 06:18:39.319972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:39.319850 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="769ed03ae5ec7b00d9034ab59f7513d29c37e5a671919d62780cf666cca4c91c" Oct 30 06:18:39.477334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:39.468413 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:18:39.506600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:39.505520 5001 patch_prober.go:29] interesting pod/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 06:18:39.506600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:39.505585 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 06:18:40.120268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:40.120230 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:18:40.120675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:40.120288 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:18:40.329866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:40.329816 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" event={"ID":"a63052f6-d65d-4307-93bb-daa7a5581e78","Type":"ContainerStarted","Data":"82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d"} Oct 30 06:18:40.392748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:40.392646 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" podStartSLOduration=3.392631527 podStartE2EDuration="3.392631527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:40.387936406 +0000 UTC m=+1534.383893413" watchObservedRunningTime="2026-10-30 06:18:40.392631527 +0000 UTC m=+1534.388588537" Oct 30 06:18:40.536335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:40.528633 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:18:40.536335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:40.534881 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:18:41.120022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:41.119983 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:18:41.120343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:41.120291 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:18:41.248194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:41.248160 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:41.273332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:41.269690 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:41.338488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:41.338456 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:41.339608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:41.339588 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:18:42.120423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:42.120394 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:18:42.120856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:42.120444 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:18:43.125742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:43.125713 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:43.126403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:43.126200 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:43.127106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:43.127085 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:18:44.886423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:44.886374 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd"] Oct 30 06:18:44.887570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:44.887543 5001 cpu_manager.go:414] "containerMap: removing container" podUID="91db3de6-a8d4-407f-86b7-80fb8076186c" containerName="revoke-keys" Oct 30 06:18:44.887570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:44.887571 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91db3de6-a8d4-407f-86b7-80fb8076186c" containerName="revoke-keys" Oct 30 06:18:44.887775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:44.887754 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="91db3de6-a8d4-407f-86b7-80fb8076186c" containerName="revoke-keys" Oct 30 06:18:44.893467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:44.893432 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd"] Oct 30 06:18:44.893601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:44.893560 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:18:45.059338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.057683 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bc5d8f-dockercfg-j9ms7\"" Oct 30 06:18:45.059338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.057805 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bc5d8f\"" Oct 30 06:18:45.087372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.084965 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1407ec9d-8fd0-4752-90ac-26b31e3d3035-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd\" (UID: \"1407ec9d-8fd0-4752-90ac-26b31e3d3035\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:18:45.087372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.085034 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w96ll\" (UniqueName: \"kubernetes.io/projected/1407ec9d-8fd0-4752-90ac-26b31e3d3035-kube-api-access-w96ll\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd\" (UID: \"1407ec9d-8fd0-4752-90ac-26b31e3d3035\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:18:45.192523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.190032 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1407ec9d-8fd0-4752-90ac-26b31e3d3035-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd\" (UID: \"1407ec9d-8fd0-4752-90ac-26b31e3d3035\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:18:45.192523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.190101 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w96ll\" (UniqueName: \"kubernetes.io/projected/1407ec9d-8fd0-4752-90ac-26b31e3d3035-kube-api-access-w96ll\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd\" (UID: \"1407ec9d-8fd0-4752-90ac-26b31e3d3035\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:18:45.230340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.194096 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1407ec9d-8fd0-4752-90ac-26b31e3d3035-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd\" (UID: \"1407ec9d-8fd0-4752-90ac-26b31e3d3035\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:18:45.257929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.257840 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w96ll\" (UniqueName: \"kubernetes.io/projected/1407ec9d-8fd0-4752-90ac-26b31e3d3035-kube-api-access-w96ll\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd\" (UID: \"1407ec9d-8fd0-4752-90ac-26b31e3d3035\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:18:45.309017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.308986 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c"] Oct 30 06:18:45.435102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.434656 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:18:45.495377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.491590 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:18:45.592907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.575892 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qbh\" (UniqueName: \"kubernetes.io/projected/aeb32039-e94a-43e6-83d2-054fcc21849e-kube-api-access-f7qbh\") pod \"payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c\" (UID: \"aeb32039-e94a-43e6-83d2-054fcc21849e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:18:45.630605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.576126 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb32039-e94a-43e6-83d2-054fcc21849e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c\" (UID: \"aeb32039-e94a-43e6-83d2-054fcc21849e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:18:45.725339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.725288 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c"] Oct 30 06:18:45.732983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.732943 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb32039-e94a-43e6-83d2-054fcc21849e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c\" (UID: \"aeb32039-e94a-43e6-83d2-054fcc21849e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:18:45.733220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.733147 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qbh\" (UniqueName: \"kubernetes.io/projected/aeb32039-e94a-43e6-83d2-054fcc21849e-kube-api-access-f7qbh\") pod \"payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c\" (UID: \"aeb32039-e94a-43e6-83d2-054fcc21849e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:18:45.734119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.734091 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb32039-e94a-43e6-83d2-054fcc21849e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c\" (UID: \"aeb32039-e94a-43e6-83d2-054fcc21849e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:18:45.734388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.734366 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44"] Oct 30 06:18:45.742149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.742082 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd"] Oct 30 06:18:45.778112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:45.778073 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qbh\" (UniqueName: \"kubernetes.io/projected/aeb32039-e94a-43e6-83d2-054fcc21849e-kube-api-access-f7qbh\") pod \"payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c\" (UID: \"aeb32039-e94a-43e6-83d2-054fcc21849e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:18:46.027388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:46.009624 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:18:46.692464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:46.692424 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f"] Oct 30 06:18:46.695591 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:18:46.695533 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1407ec9d_8fd0_4752_90ac_26b31e3d3035.slice/crio-c756c4f54670a197d91f73e95f31bcd67ea60237e15d5304d98582d0e7ab3934 WatchSource:0}: Error finding container c756c4f54670a197d91f73e95f31bcd67ea60237e15d5304d98582d0e7ab3934: Status 404 returned error can't find the container with id c756c4f54670a197d91f73e95f31bcd67ea60237e15d5304d98582d0e7ab3934 Oct 30 06:18:46.775644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:46.748467 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 06:18:46.805561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:46.805082 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j"] Oct 30 06:18:46.867158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:46.867065 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd"] Oct 30 06:18:47.278348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:47.254094 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c"] Oct 30 06:18:47.894316 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:47.894277 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" event={"ID":"1407ec9d-8fd0-4752-90ac-26b31e3d3035","Type":"ContainerStarted","Data":"356262877e35f0271de7fed51e17ca4cee24cf876a26c723d7838cb672f5aef9"} Oct 30 06:18:47.894536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:47.894373 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" event={"ID":"1407ec9d-8fd0-4752-90ac-26b31e3d3035","Type":"ContainerStarted","Data":"c756c4f54670a197d91f73e95f31bcd67ea60237e15d5304d98582d0e7ab3934"} Oct 30 06:18:47.894536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:47.894443 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:18:47.896527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:47.896506 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" event={"ID":"aeb32039-e94a-43e6-83d2-054fcc21849e","Type":"ContainerStarted","Data":"7685a8fdb4ecc0f530f0e390bd2a722a9d7e90de1a1899a0e6a1fefd01353f9a"} Oct 30 06:18:47.896651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:47.896542 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" event={"ID":"aeb32039-e94a-43e6-83d2-054fcc21849e","Type":"ContainerStarted","Data":"b62cfdddf6b9de8077ffccac52ebe6a127b5be69a9997fff3741fe9eeead81a8"} Oct 30 06:18:47.896651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:47.896585 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:18:47.939808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:47.939756 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" podStartSLOduration=3.939744215 podStartE2EDuration="3.939744215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:47.93529446 +0000 UTC m=+1541.931251471" watchObservedRunningTime="2026-10-30 06:18:47.939744215 +0000 UTC m=+1541.935701275" Oct 30 06:18:47.986893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:47.986826 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" podStartSLOduration=3.986804378 podStartE2EDuration="3.986804378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:47.974647761 +0000 UTC m=+1541.970604785" watchObservedRunningTime="2026-10-30 06:18:47.986804378 +0000 UTC m=+1541.982761388" Oct 30 06:18:49.778094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.778051 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96"] Oct 30 06:18:49.784025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.783997 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.821438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.819030 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.821438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.819087 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.821438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.819234 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5kv9\" (UniqueName: \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-kube-api-access-h5kv9\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.821438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.819281 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.821438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.819380 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.821438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.819404 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.821438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.819419 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.821438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.819452 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.821438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.819491 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.821438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.820648 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96"] Oct 30 06:18:49.920699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.920666 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.920877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.920720 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.920877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.920838 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.921004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.920877 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.921004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.920971 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5kv9\" (UniqueName: \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-kube-api-access-h5kv9\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.921113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.921009 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.921113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.921054 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.921288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.921125 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.921288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.921161 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.921288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.921188 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.921553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.921286 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.921553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.921518 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.921679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.921647 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.922417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.922394 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.923949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.923926 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.924278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.924260 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.943872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.943843 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:49.944018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:49.943997 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5kv9\" (UniqueName: \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-kube-api-access-h5kv9\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:50.103553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.103457 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:50.464101 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:18:50.464070 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d0d063_92c4_4dd1_a1d4_a546bbd47eb6.slice/crio-6d5b4323feb55ba97bad6a0dbdea6ac443746afc029e7161e74c95b7697ac060 WatchSource:0}: Error finding container 6d5b4323feb55ba97bad6a0dbdea6ac443746afc029e7161e74c95b7697ac060: Status 404 returned error can't find the container with id 6d5b4323feb55ba97bad6a0dbdea6ac443746afc029e7161e74c95b7697ac060 Oct 30 06:18:50.465001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.464976 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96"] Oct 30 06:18:50.467647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.467593 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:50.467770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.467679 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:50.467770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.467712 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:50.735266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.735164 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" podUID="58b8efdd-c28a-4818-a8a5-fbd3a422669e" containerName="payload-processing" containerID="cri-o://8e73d25e7abc17d49bca2821642d6170c7bcdc82753f31da260fd04041ca8068" gracePeriod=25 Oct 30 06:18:50.743254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.743212 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" podUID="f3b400d0-7c38-4d95-8a56-7fdd98a39f5a" containerName="payload-pre-processing" containerID="cri-o://2f80ca4b9d1c86cfa3fd5ea912aa3af273f31c13e2d2e1d3b98c097d4ed9b825" gracePeriod=25 Oct 30 06:18:50.932570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.932155 5001 generic.go:356] "Generic (PLEG): container finished" podID="58b8efdd-c28a-4818-a8a5-fbd3a422669e" containerID="8e73d25e7abc17d49bca2821642d6170c7bcdc82753f31da260fd04041ca8068" exitCode=0 Oct 30 06:18:50.932570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.932424 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" event={"ID":"58b8efdd-c28a-4818-a8a5-fbd3a422669e","Type":"ContainerDied","Data":"8e73d25e7abc17d49bca2821642d6170c7bcdc82753f31da260fd04041ca8068"} Oct 30 06:18:50.936837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.935539 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" event={"ID":"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6","Type":"ContainerStarted","Data":"c58fe15738a9549ed62b53893c4269a3f887967c028deac887d75d6db3e81830"} Oct 30 06:18:50.936837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.935599 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" event={"ID":"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6","Type":"ContainerStarted","Data":"6d5b4323feb55ba97bad6a0dbdea6ac443746afc029e7161e74c95b7697ac060"} Oct 30 06:18:50.940260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.940075 5001 generic.go:356] "Generic (PLEG): container finished" podID="f3b400d0-7c38-4d95-8a56-7fdd98a39f5a" containerID="2f80ca4b9d1c86cfa3fd5ea912aa3af273f31c13e2d2e1d3b98c097d4ed9b825" exitCode=0 Oct 30 06:18:50.940260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:50.940162 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" event={"ID":"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a","Type":"ContainerDied","Data":"2f80ca4b9d1c86cfa3fd5ea912aa3af273f31c13e2d2e1d3b98c097d4ed9b825"} Oct 30 06:18:51.018291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.018148 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" podStartSLOduration=2.018125915 podStartE2EDuration="2.018125915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:51.008392218 +0000 UTC m=+1545.004349227" watchObservedRunningTime="2026-10-30 06:18:51.018125915 +0000 UTC m=+1545.014082926" Oct 30 06:18:51.104002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.103975 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:51.105412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.105231 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:18:51.105501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.105432 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:18:51.112609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.112586 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:18:51.144841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.144807 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b8efdd-c28a-4818-a8a5-fbd3a422669e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b8efdd-c28a-4818-a8a5-fbd3a422669e-plugins-config-volume\") pod \"58b8efdd-c28a-4818-a8a5-fbd3a422669e\" (UID: \"58b8efdd-c28a-4818-a8a5-fbd3a422669e\") " Oct 30 06:18:51.145017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.144857 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b8efdd-c28a-4818-a8a5-fbd3a422669e-kube-api-access-bkttk\" (UniqueName: \"kubernetes.io/projected/58b8efdd-c28a-4818-a8a5-fbd3a422669e-kube-api-access-bkttk\") pod \"58b8efdd-c28a-4818-a8a5-fbd3a422669e\" (UID: \"58b8efdd-c28a-4818-a8a5-fbd3a422669e\") " Oct 30 06:18:51.145273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.145250 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b8efdd-c28a-4818-a8a5-fbd3a422669e-plugins-config-volume" pod "58b8efdd-c28a-4818-a8a5-fbd3a422669e" (UID: "58b8efdd-c28a-4818-a8a5-fbd3a422669e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:51.145551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.145528 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b8efdd-c28a-4818-a8a5-fbd3a422669e-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:51.147600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.147573 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b8efdd-c28a-4818-a8a5-fbd3a422669e-kube-api-access-bkttk" pod "58b8efdd-c28a-4818-a8a5-fbd3a422669e" (UID: "58b8efdd-c28a-4818-a8a5-fbd3a422669e"). InnerVolumeSpecName "kube-api-access-bkttk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:51.192207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.192170 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:18:51.300297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.282378 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-kube-api-access-5pv6h\" (UniqueName: \"kubernetes.io/projected/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-kube-api-access-5pv6h\") pod \"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a\" (UID: \"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a\") " Oct 30 06:18:51.306803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.306712 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-kube-api-access-5pv6h" pod "f3b400d0-7c38-4d95-8a56-7fdd98a39f5a" (UID: "f3b400d0-7c38-4d95-8a56-7fdd98a39f5a"). InnerVolumeSpecName "kube-api-access-5pv6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:51.307407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.307373 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-plugins-config-volume\") pod \"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a\" (UID: \"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a\") " Oct 30 06:18:51.308369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.307918 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-plugins-config-volume" pod "f3b400d0-7c38-4d95-8a56-7fdd98a39f5a" (UID: "f3b400d0-7c38-4d95-8a56-7fdd98a39f5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:51.308369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.308293 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkttk\" (UniqueName: \"kubernetes.io/projected/58b8efdd-c28a-4818-a8a5-fbd3a422669e-kube-api-access-bkttk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:51.308369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.308330 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pv6h\" (UniqueName: \"kubernetes.io/projected/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-kube-api-access-5pv6h\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:51.308369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.308347 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:51.694758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.694705 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" podUID="a5bda487-9e0e-40da-8340-7577ab18ca42" containerName="payload-processing" containerID="cri-o://e34e55f7b2f6c1337d66312e91ee86e3cf66bd32157fe12ff95bb1d8a8e6c947" gracePeriod=25 Oct 30 06:18:51.825780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:51.825684 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" podUID="ddcce932-f11c-45af-b2a8-3563da5b90ac" containerName="payload-pre-processing" containerID="cri-o://1aaf702316d6eeb1e1e71ce52a9aad979f368cec5d4e317f6376cb36f1cf9431" gracePeriod=25 Oct 30 06:18:52.005402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.002002 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" Oct 30 06:18:52.005402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.003426 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd" event={"ID":"f3b400d0-7c38-4d95-8a56-7fdd98a39f5a","Type":"ContainerDied","Data":"e75cc44865e097563633585a4058dc8edc2d41fd460bf8f38a2c7b4cf194c3d5"} Oct 30 06:18:52.005402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.003490 5001 scope.go:122] "RemoveContainer" containerID="2f80ca4b9d1c86cfa3fd5ea912aa3af273f31c13e2d2e1d3b98c097d4ed9b825" Oct 30 06:18:52.009544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.009456 5001 generic.go:356] "Generic (PLEG): container finished" podID="ddcce932-f11c-45af-b2a8-3563da5b90ac" containerID="1aaf702316d6eeb1e1e71ce52a9aad979f368cec5d4e317f6376cb36f1cf9431" exitCode=0 Oct 30 06:18:52.009628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.009538 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" event={"ID":"ddcce932-f11c-45af-b2a8-3563da5b90ac","Type":"ContainerDied","Data":"1aaf702316d6eeb1e1e71ce52a9aad979f368cec5d4e317f6376cb36f1cf9431"} Oct 30 06:18:52.016633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.016112 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" event={"ID":"58b8efdd-c28a-4818-a8a5-fbd3a422669e","Type":"ContainerDied","Data":"d4169824df00c0019b1badeb97c2d35a1cd6673490b38475321ed526f1530563"} Oct 30 06:18:52.016633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.016254 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44" Oct 30 06:18:52.023070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.022101 5001 generic.go:356] "Generic (PLEG): container finished" podID="a5bda487-9e0e-40da-8340-7577ab18ca42" containerID="e34e55f7b2f6c1337d66312e91ee86e3cf66bd32157fe12ff95bb1d8a8e6c947" exitCode=0 Oct 30 06:18:52.023070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.022204 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" event={"ID":"a5bda487-9e0e-40da-8340-7577ab18ca42","Type":"ContainerDied","Data":"e34e55f7b2f6c1337d66312e91ee86e3cf66bd32157fe12ff95bb1d8a8e6c947"} Oct 30 06:18:52.097366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.097334 5001 scope.go:122] "RemoveContainer" containerID="8e73d25e7abc17d49bca2821642d6170c7bcdc82753f31da260fd04041ca8068" Oct 30 06:18:52.099926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.099847 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:18:52.105412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.105357 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:18:52.106439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.106100 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:18:52.127038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.127009 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd"] Oct 30 06:18:52.131498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.131473 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0fab6d-5b4ffc9499-d8msd"] Oct 30 06:18:52.210437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.210356 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44"] Oct 30 06:18:52.217955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.217883 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0fab6d-798fc6fdb4-82s44"] Oct 30 06:18:52.225789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.224846 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bda487-9e0e-40da-8340-7577ab18ca42-kube-api-access-gh26p\" (UniqueName: \"kubernetes.io/projected/a5bda487-9e0e-40da-8340-7577ab18ca42-kube-api-access-gh26p\") pod \"a5bda487-9e0e-40da-8340-7577ab18ca42\" (UID: \"a5bda487-9e0e-40da-8340-7577ab18ca42\") " Oct 30 06:18:52.225789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.224911 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5bda487-9e0e-40da-8340-7577ab18ca42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bda487-9e0e-40da-8340-7577ab18ca42-plugins-config-volume\") pod \"a5bda487-9e0e-40da-8340-7577ab18ca42\" (UID: \"a5bda487-9e0e-40da-8340-7577ab18ca42\") " Oct 30 06:18:52.226373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.226090 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bda487-9e0e-40da-8340-7577ab18ca42-plugins-config-volume" pod "a5bda487-9e0e-40da-8340-7577ab18ca42" (UID: "a5bda487-9e0e-40da-8340-7577ab18ca42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:52.228218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.228191 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bda487-9e0e-40da-8340-7577ab18ca42-kube-api-access-gh26p" pod "a5bda487-9e0e-40da-8340-7577ab18ca42" (UID: "a5bda487-9e0e-40da-8340-7577ab18ca42"). InnerVolumeSpecName "kube-api-access-gh26p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:52.327526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.327479 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh26p\" (UniqueName: \"kubernetes.io/projected/a5bda487-9e0e-40da-8340-7577ab18ca42-kube-api-access-gh26p\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:52.327526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.327523 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bda487-9e0e-40da-8340-7577ab18ca42-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:52.330379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.329419 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:18:52.428677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.428647 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddcce932-f11c-45af-b2a8-3563da5b90ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcce932-f11c-45af-b2a8-3563da5b90ac-plugins-config-volume\") pod \"ddcce932-f11c-45af-b2a8-3563da5b90ac\" (UID: \"ddcce932-f11c-45af-b2a8-3563da5b90ac\") " Oct 30 06:18:52.428829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.428681 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddcce932-f11c-45af-b2a8-3563da5b90ac-kube-api-access-6h2wt\" (UniqueName: \"kubernetes.io/projected/ddcce932-f11c-45af-b2a8-3563da5b90ac-kube-api-access-6h2wt\") pod \"ddcce932-f11c-45af-b2a8-3563da5b90ac\" (UID: \"ddcce932-f11c-45af-b2a8-3563da5b90ac\") " Oct 30 06:18:52.430026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.429944 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddcce932-f11c-45af-b2a8-3563da5b90ac-plugins-config-volume" pod "ddcce932-f11c-45af-b2a8-3563da5b90ac" (UID: "ddcce932-f11c-45af-b2a8-3563da5b90ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:52.431661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.431636 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddcce932-f11c-45af-b2a8-3563da5b90ac-kube-api-access-6h2wt" pod "ddcce932-f11c-45af-b2a8-3563da5b90ac" (UID: "ddcce932-f11c-45af-b2a8-3563da5b90ac"). InnerVolumeSpecName "kube-api-access-6h2wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:52.530660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.530622 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcce932-f11c-45af-b2a8-3563da5b90ac-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:52.530660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:52.530654 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2wt\" (UniqueName: \"kubernetes.io/projected/ddcce932-f11c-45af-b2a8-3563da5b90ac-kube-api-access-6h2wt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:18:53.033689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.033660 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" Oct 30 06:18:53.034170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.033711 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j" event={"ID":"ddcce932-f11c-45af-b2a8-3563da5b90ac","Type":"ContainerDied","Data":"54a5cc137323b3e5aa76a4dedb36ff23a1242bc01be7b6dfc8ced8204b79c71b"} Oct 30 06:18:53.034170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.033770 5001 scope.go:122] "RemoveContainer" containerID="1aaf702316d6eeb1e1e71ce52a9aad979f368cec5d4e317f6376cb36f1cf9431" Oct 30 06:18:53.036933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.036901 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" event={"ID":"a5bda487-9e0e-40da-8340-7577ab18ca42","Type":"ContainerDied","Data":"3646c635a906878222c25c0089eef1090ac0d69e575015f0bd52d808ebf17a8e"} Oct 30 06:18:53.037079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.036978 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f" Oct 30 06:18:53.047998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.047972 5001 scope.go:122] "RemoveContainer" containerID="e34e55f7b2f6c1337d66312e91ee86e3cf66bd32157fe12ff95bb1d8a8e6c947" Oct 30 06:18:53.140579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.114236 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:18:53.140579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.114296 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:18:53.140579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.125893 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j"] Oct 30 06:18:53.164342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.151809 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-f7756d44c-zql6j"] Oct 30 06:18:53.243383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.242681 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f"] Oct 30 06:18:53.247974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:53.247935 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-bf7fb9ddb-ddn7f"] Oct 30 06:18:54.104639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:54.104599 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:18:54.105262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:54.104652 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:18:54.118095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:54.118064 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b8efdd-c28a-4818-a8a5-fbd3a422669e" path="/var/lib/kubelet/pods/58b8efdd-c28a-4818-a8a5-fbd3a422669e/volumes" Oct 30 06:18:54.118572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:54.118551 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bda487-9e0e-40da-8340-7577ab18ca42" path="/var/lib/kubelet/pods/a5bda487-9e0e-40da-8340-7577ab18ca42/volumes" Oct 30 06:18:54.118998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:54.118979 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddcce932-f11c-45af-b2a8-3563da5b90ac" path="/var/lib/kubelet/pods/ddcce932-f11c-45af-b2a8-3563da5b90ac/volumes" Oct 30 06:18:54.119432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:54.119414 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b400d0-7c38-4d95-8a56-7fdd98a39f5a" path="/var/lib/kubelet/pods/f3b400d0-7c38-4d95-8a56-7fdd98a39f5a/volumes" Oct 30 06:18:55.111446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:55.111011 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:18:55.111446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:55.111068 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:18:56.091159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:56.091127 5001 generic.go:356] "Generic (PLEG): container finished" podID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerID="52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9" exitCode=0 Oct 30 06:18:56.091363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:56.091203 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" event={"ID":"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1","Type":"ContainerDied","Data":"52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9"} Oct 30 06:18:56.091658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:56.091644 5001 scope.go:122] "RemoveContainer" containerID="52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9" Oct 30 06:18:56.093601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:56.093567 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:56.093685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:56.093665 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:56.093727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:56.093714 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:56.107535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:56.107495 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:18:56.107691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:56.107568 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:18:57.106159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.106111 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:18:57.106806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.106753 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:18:57.117112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.117083 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j"] Oct 30 06:18:57.127886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.127835 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" podUID="abd10d8d-70a5-4897-81c8-0875ff56a289" containerName="istio-proxy" containerID="cri-o://93d6986eb5bb60ad129b6a6be3d0c7c16a9fe3dc34d4a1991056ce72c46519c6" gracePeriod=30 Oct 30 06:18:57.137366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.136406 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf"] Oct 30 06:18:57.137366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.136666 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" podUID="fd447616-cb5a-417a-994f-25acea127f8f" containerName="istio-proxy" containerID="cri-o://65347044dfc1dcf85109df612336f773f80e88e56c168b77da2f49ca52d7f754" gracePeriod=30 Oct 30 06:18:57.141274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.138629 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" event={"ID":"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1","Type":"ContainerStarted","Data":"e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd"} Oct 30 06:18:57.144650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.143413 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl"] Oct 30 06:18:57.144650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.143611 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" podUID="1c7ca03e-63bc-4107-85aa-73852f18cf40" containerName="istio-proxy" containerID="cri-o://2ed43fa7a8dc76c8ba309717ded505c1ba20653e60821f6ffffcecd849e6f6c8" gracePeriod=30 Oct 30 06:18:57.154660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.154632 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh"] Oct 30 06:18:57.155066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.155040 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" containerID="cri-o://f99f9c26fbb087955cb3c18faeb4002b832f2982bd1335137205e2ced33b6eb8" gracePeriod=30 Oct 30 06:18:57.462899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.462755 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:18:57.466100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.466057 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 06:18:57.466656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:57.466624 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 06:18:58.122056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:58.122027 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:58.122804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:58.122783 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:58.123551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:58.123529 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:18:58.489394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:58.485920 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 06:18:58.489394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:58.485978 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 06:18:58.944345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:58.943650 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:18:58.948335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:58.944923 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:18:59.464726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:59.464685 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 06:18:59.465407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:18:59.465371 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 06:19:00.466336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:00.463767 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 06:19:00.466336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:00.463855 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 06:19:00.906664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:00.906623 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx"] Oct 30 06:19:00.917145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:00.917109 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm"] Oct 30 06:19:01.464176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:01.463702 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 06:19:01.464176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:01.463779 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 06:19:02.507331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:02.480441 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 06:19:02.507331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:02.480506 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 06:19:02.624013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:02.623976 5001 generic.go:356] "Generic (PLEG): container finished" podID="fd447616-cb5a-417a-994f-25acea127f8f" containerID="65347044dfc1dcf85109df612336f773f80e88e56c168b77da2f49ca52d7f754" exitCode=0 Oct 30 06:19:02.624170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:02.624076 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" event={"ID":"fd447616-cb5a-417a-994f-25acea127f8f","Type":"ContainerDied","Data":"65347044dfc1dcf85109df612336f773f80e88e56c168b77da2f49ca52d7f754"} Oct 30 06:19:02.626706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:02.626513 5001 generic.go:356] "Generic (PLEG): container finished" podID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerID="f99f9c26fbb087955cb3c18faeb4002b832f2982bd1335137205e2ced33b6eb8" exitCode=0 Oct 30 06:19:02.626706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:02.626626 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" event={"ID":"69af5daa-6e0f-416c-b808-8a2ef5eaa603","Type":"ContainerDied","Data":"f99f9c26fbb087955cb3c18faeb4002b832f2982bd1335137205e2ced33b6eb8"} Oct 30 06:19:02.629242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:02.629211 5001 generic.go:356] "Generic (PLEG): container finished" podID="1c7ca03e-63bc-4107-85aa-73852f18cf40" containerID="2ed43fa7a8dc76c8ba309717ded505c1ba20653e60821f6ffffcecd849e6f6c8" exitCode=0 Oct 30 06:19:02.629517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:02.629494 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" event={"ID":"1c7ca03e-63bc-4107-85aa-73852f18cf40","Type":"ContainerDied","Data":"2ed43fa7a8dc76c8ba309717ded505c1ba20653e60821f6ffffcecd849e6f6c8"} Oct 30 06:19:02.632446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:02.632324 5001 generic.go:356] "Generic (PLEG): container finished" podID="abd10d8d-70a5-4897-81c8-0875ff56a289" containerID="93d6986eb5bb60ad129b6a6be3d0c7c16a9fe3dc34d4a1991056ce72c46519c6" exitCode=0 Oct 30 06:19:02.632446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:02.632373 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" event={"ID":"abd10d8d-70a5-4897-81c8-0875ff56a289","Type":"ContainerDied","Data":"93d6986eb5bb60ad129b6a6be3d0c7c16a9fe3dc34d4a1991056ce72c46519c6"} Oct 30 06:19:02.750441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:02.729775 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:19:03.009565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.004094 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:19:03.085337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.081815 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-socket\") pod \"abd10d8d-70a5-4897-81c8-0875ff56a289\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " Oct 30 06:19:03.085337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.081868 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-kube-api-access-8zp64\" (UniqueName: \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-kube-api-access-8zp64\") pod \"abd10d8d-70a5-4897-81c8-0875ff56a289\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " Oct 30 06:19:03.085337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.081930 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-envoy\") pod \"abd10d8d-70a5-4897-81c8-0875ff56a289\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " Oct 30 06:19:03.085337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.081965 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-token\" (UniqueName: \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-token\") pod \"abd10d8d-70a5-4897-81c8-0875ff56a289\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " Oct 30 06:19:03.085337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.082143 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-certs\") pod \"abd10d8d-70a5-4897-81c8-0875ff56a289\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " Oct 30 06:19:03.085337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.082197 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd10d8d-70a5-4897-81c8-0875ff56a289-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd10d8d-70a5-4897-81c8-0875ff56a289-istiod-ca-cert\") pod \"abd10d8d-70a5-4897-81c8-0875ff56a289\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " Oct 30 06:19:03.085337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.082274 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-data\") pod \"abd10d8d-70a5-4897-81c8-0875ff56a289\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " Oct 30 06:19:03.144255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.143883 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-data" pod "abd10d8d-70a5-4897-81c8-0875ff56a289" (UID: "abd10d8d-70a5-4897-81c8-0875ff56a289"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:03.178704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.144623 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-podinfo\") pod \"abd10d8d-70a5-4897-81c8-0875ff56a289\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " Oct 30 06:19:03.178704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.144754 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-credential-socket\") pod \"abd10d8d-70a5-4897-81c8-0875ff56a289\" (UID: \"abd10d8d-70a5-4897-81c8-0875ff56a289\") " Oct 30 06:19:03.178704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.148603 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd10d8d-70a5-4897-81c8-0875ff56a289-istiod-ca-cert" pod "abd10d8d-70a5-4897-81c8-0875ff56a289" (UID: "abd10d8d-70a5-4897-81c8-0875ff56a289"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:03.178704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.148939 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-certs" pod "abd10d8d-70a5-4897-81c8-0875ff56a289" (UID: "abd10d8d-70a5-4897-81c8-0875ff56a289"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:03.178704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.149355 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-credential-socket" pod "abd10d8d-70a5-4897-81c8-0875ff56a289" (UID: "abd10d8d-70a5-4897-81c8-0875ff56a289"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:03.178704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.170824 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.178704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.170856 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd10d8d-70a5-4897-81c8-0875ff56a289-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.178704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.170871 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.178704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.170884 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.178704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.174430 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-socket" pod "abd10d8d-70a5-4897-81c8-0875ff56a289" (UID: "abd10d8d-70a5-4897-81c8-0875ff56a289"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:03.229441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.229369 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-kube-api-access-8zp64" pod "abd10d8d-70a5-4897-81c8-0875ff56a289" (UID: "abd10d8d-70a5-4897-81c8-0875ff56a289"). InnerVolumeSpecName "kube-api-access-8zp64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:03.231479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.231357 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-podinfo" pod "abd10d8d-70a5-4897-81c8-0875ff56a289" (UID: "abd10d8d-70a5-4897-81c8-0875ff56a289"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:19:03.331290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.290717 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.331290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.290753 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.331290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.290776 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zp64\" (UniqueName: \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-kube-api-access-8zp64\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.331290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.317617 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-envoy" pod "abd10d8d-70a5-4897-81c8-0875ff56a289" (UID: "abd10d8d-70a5-4897-81c8-0875ff56a289"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:03.331290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.330695 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-token" pod "abd10d8d-70a5-4897-81c8-0875ff56a289" (UID: "abd10d8d-70a5-4897-81c8-0875ff56a289"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:03.400512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.397517 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.400512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.397554 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd10d8d-70a5-4897-81c8-0875ff56a289-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.447373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.445601 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:19:03.551113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.551077 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 06:19:03.597421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.573750 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 06:19:03.632778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.632701 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-podinfo\") pod \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " Oct 30 06:19:03.649333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.637432 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-token\" (UniqueName: \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-token\") pod \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " Oct 30 06:19:03.650532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.650499 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-data\") pod \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " Oct 30 06:19:03.650740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.650721 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-envoy\") pod \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " Oct 30 06:19:03.650941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.650918 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-kube-api-access-wkzr6\" (UniqueName: \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-kube-api-access-wkzr6\") pod \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " Oct 30 06:19:03.651151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.651135 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-socket\") pod \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " Oct 30 06:19:03.651336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.651292 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-credential-socket\") pod \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " Oct 30 06:19:03.651492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.651471 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-certs\") pod \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " Oct 30 06:19:03.651632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.651617 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istiod-ca-cert\") pod \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\" (UID: \"69af5daa-6e0f-416c-b808-8a2ef5eaa603\") " Oct 30 06:19:03.671592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.671546 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istiod-ca-cert" pod "69af5daa-6e0f-416c-b808-8a2ef5eaa603" (UID: "69af5daa-6e0f-416c-b808-8a2ef5eaa603"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:03.679594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.675708 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-data" pod "69af5daa-6e0f-416c-b808-8a2ef5eaa603" (UID: "69af5daa-6e0f-416c-b808-8a2ef5eaa603"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:03.686925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.686880 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-socket" pod "69af5daa-6e0f-416c-b808-8a2ef5eaa603" (UID: "69af5daa-6e0f-416c-b808-8a2ef5eaa603"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:03.692459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.692417 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-credential-socket" pod "69af5daa-6e0f-416c-b808-8a2ef5eaa603" (UID: "69af5daa-6e0f-416c-b808-8a2ef5eaa603"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:03.713429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.702818 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-certs" pod "69af5daa-6e0f-416c-b808-8a2ef5eaa603" (UID: "69af5daa-6e0f-416c-b808-8a2ef5eaa603"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:03.734456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.734408 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-token" pod "69af5daa-6e0f-416c-b808-8a2ef5eaa603" (UID: "69af5daa-6e0f-416c-b808-8a2ef5eaa603"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:03.793331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.792660 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.793331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.792696 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.793331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.792711 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.793331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.792726 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.793331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.792739 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.793331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.792751 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.804549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.799478 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" Oct 30 06:19:03.804549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.799886 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh" event={"ID":"69af5daa-6e0f-416c-b808-8a2ef5eaa603","Type":"ContainerDied","Data":"281f4be85bdbe874a34d7047a37f792e170bdbe2e7cdc9a22f2eb273a3a23128"} Oct 30 06:19:03.804549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.799949 5001 scope.go:122] "RemoveContainer" containerID="f99f9c26fbb087955cb3c18faeb4002b832f2982bd1335137205e2ced33b6eb8" Oct 30 06:19:03.804549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.803059 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" event={"ID":"abd10d8d-70a5-4897-81c8-0875ff56a289","Type":"ContainerDied","Data":"3618d78b3cb2430504b681fe5b663e47c5c22cdcc8a363093851a724fce01e9c"} Oct 30 06:19:03.804549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.803173 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j" Oct 30 06:19:03.822053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.822020 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-envoy" pod "69af5daa-6e0f-416c-b808-8a2ef5eaa603" (UID: "69af5daa-6e0f-416c-b808-8a2ef5eaa603"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:03.822337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.822264 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-podinfo" pod "69af5daa-6e0f-416c-b808-8a2ef5eaa603" (UID: "69af5daa-6e0f-416c-b808-8a2ef5eaa603"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:19:03.823860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.823831 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-kube-api-access-wkzr6" pod "69af5daa-6e0f-416c-b808-8a2ef5eaa603" (UID: "69af5daa-6e0f-416c-b808-8a2ef5eaa603"). InnerVolumeSpecName "kube-api-access-wkzr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:03.826019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.825996 5001 scope.go:122] "RemoveContainer" containerID="93d6986eb5bb60ad129b6a6be3d0c7c16a9fe3dc34d4a1991056ce72c46519c6" Oct 30 06:19:03.834329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.834203 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:19:03.851403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.851278 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:19:03.894699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.894658 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.894699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.894700 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzr6\" (UniqueName: \"kubernetes.io/projected/69af5daa-6e0f-416c-b808-8a2ef5eaa603-kube-api-access-wkzr6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.894882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.894716 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69af5daa-6e0f-416c-b808-8a2ef5eaa603-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:03.896720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.896699 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j"] Oct 30 06:19:03.903799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.903770 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pnd9j"] Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.996911 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-credential-socket\") pod \"fd447616-cb5a-417a-994f-25acea127f8f\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.996966 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-token\" (UniqueName: \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-token\") pod \"1c7ca03e-63bc-4107-85aa-73852f18cf40\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997012 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-socket\") pod \"fd447616-cb5a-417a-994f-25acea127f8f\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997050 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-kube-api-access-8znsm\" (UniqueName: \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-kube-api-access-8znsm\") pod \"fd447616-cb5a-417a-994f-25acea127f8f\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997116 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-envoy\") pod \"1c7ca03e-63bc-4107-85aa-73852f18cf40\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997176 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-credential-socket\") pod \"1c7ca03e-63bc-4107-85aa-73852f18cf40\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997214 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-istio-token\" (UniqueName: \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-istio-token\") pod \"fd447616-cb5a-417a-994f-25acea127f8f\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997247 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-data\") pod \"fd447616-cb5a-417a-994f-25acea127f8f\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997402 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd447616-cb5a-417a-994f-25acea127f8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd447616-cb5a-417a-994f-25acea127f8f-istiod-ca-cert\") pod \"fd447616-cb5a-417a-994f-25acea127f8f\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997447 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7ca03e-63bc-4107-85aa-73852f18cf40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7ca03e-63bc-4107-85aa-73852f18cf40-istiod-ca-cert\") pod \"1c7ca03e-63bc-4107-85aa-73852f18cf40\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997498 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-kube-api-access-nbcr6\" (UniqueName: \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-kube-api-access-nbcr6\") pod \"1c7ca03e-63bc-4107-85aa-73852f18cf40\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997539 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-certs\") pod \"1c7ca03e-63bc-4107-85aa-73852f18cf40\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997594 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-podinfo\") pod \"1c7ca03e-63bc-4107-85aa-73852f18cf40\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997625 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd447616-cb5a-417a-994f-25acea127f8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd447616-cb5a-417a-994f-25acea127f8f-istio-podinfo\") pod \"fd447616-cb5a-417a-994f-25acea127f8f\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997689 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-data\") pod \"1c7ca03e-63bc-4107-85aa-73852f18cf40\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997721 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-certs\") pod \"fd447616-cb5a-417a-994f-25acea127f8f\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997800 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-socket\") pod \"1c7ca03e-63bc-4107-85aa-73852f18cf40\" (UID: \"1c7ca03e-63bc-4107-85aa-73852f18cf40\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:03.997867 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-envoy\") pod \"fd447616-cb5a-417a-994f-25acea127f8f\" (UID: \"fd447616-cb5a-417a-994f-25acea127f8f\") " Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.001632 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-credential-socket" pod "fd447616-cb5a-417a-994f-25acea127f8f" (UID: "fd447616-cb5a-417a-994f-25acea127f8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.005653 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-envoy" pod "fd447616-cb5a-417a-994f-25acea127f8f" (UID: "fd447616-cb5a-417a-994f-25acea127f8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.005973 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-certs" pod "1c7ca03e-63bc-4107-85aa-73852f18cf40" (UID: "1c7ca03e-63bc-4107-85aa-73852f18cf40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.005990 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7ca03e-63bc-4107-85aa-73852f18cf40-istiod-ca-cert" pod "1c7ca03e-63bc-4107-85aa-73852f18cf40" (UID: "1c7ca03e-63bc-4107-85aa-73852f18cf40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.006202 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-socket" pod "fd447616-cb5a-417a-994f-25acea127f8f" (UID: "fd447616-cb5a-417a-994f-25acea127f8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.007741 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-certs" pod "fd447616-cb5a-417a-994f-25acea127f8f" (UID: "fd447616-cb5a-417a-994f-25acea127f8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.007972 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-data" pod "1c7ca03e-63bc-4107-85aa-73852f18cf40" (UID: "1c7ca03e-63bc-4107-85aa-73852f18cf40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.008194 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-socket" pod "1c7ca03e-63bc-4107-85aa-73852f18cf40" (UID: "1c7ca03e-63bc-4107-85aa-73852f18cf40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.009368 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-token" pod "1c7ca03e-63bc-4107-85aa-73852f18cf40" (UID: "1c7ca03e-63bc-4107-85aa-73852f18cf40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.010163 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-data" pod "fd447616-cb5a-417a-994f-25acea127f8f" (UID: "fd447616-cb5a-417a-994f-25acea127f8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.010519 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd447616-cb5a-417a-994f-25acea127f8f-istiod-ca-cert" pod "fd447616-cb5a-417a-994f-25acea127f8f" (UID: "fd447616-cb5a-417a-994f-25acea127f8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.010619 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-credential-socket" pod "1c7ca03e-63bc-4107-85aa-73852f18cf40" (UID: "1c7ca03e-63bc-4107-85aa-73852f18cf40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.012411 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.012436 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.012452 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.012465 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.012484 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7ca03e-63bc-4107-85aa-73852f18cf40-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.012497 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.012509 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.012522 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.012540 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.012552 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd447616-cb5a-417a-994f-25acea127f8f-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.036466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.025483 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-istio-token" pod "fd447616-cb5a-417a-994f-25acea127f8f" (UID: "fd447616-cb5a-417a-994f-25acea127f8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:04.042663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.042623 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-kube-api-access-8znsm" pod "fd447616-cb5a-417a-994f-25acea127f8f" (UID: "fd447616-cb5a-417a-994f-25acea127f8f"). InnerVolumeSpecName "kube-api-access-8znsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:04.042857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.042628 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd447616-cb5a-417a-994f-25acea127f8f-istio-podinfo" pod "fd447616-cb5a-417a-994f-25acea127f8f" (UID: "fd447616-cb5a-417a-994f-25acea127f8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:19:04.042944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.042748 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-kube-api-access-nbcr6" pod "1c7ca03e-63bc-4107-85aa-73852f18cf40" (UID: "1c7ca03e-63bc-4107-85aa-73852f18cf40"). InnerVolumeSpecName "kube-api-access-nbcr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:04.044412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.044380 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-podinfo" pod "1c7ca03e-63bc-4107-85aa-73852f18cf40" (UID: "1c7ca03e-63bc-4107-85aa-73852f18cf40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:19:04.044412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.044401 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-envoy" pod "1c7ca03e-63bc-4107-85aa-73852f18cf40" (UID: "1c7ca03e-63bc-4107-85aa-73852f18cf40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:04.113878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.113851 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8znsm\" (UniqueName: \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-kube-api-access-8znsm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.114001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.113883 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.114001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.113898 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7ca03e-63bc-4107-85aa-73852f18cf40-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.114001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.113910 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd447616-cb5a-417a-994f-25acea127f8f-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.114001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.113923 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd447616-cb5a-417a-994f-25acea127f8f-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.114001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.113934 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbcr6\" (UniqueName: \"kubernetes.io/projected/1c7ca03e-63bc-4107-85aa-73852f18cf40-kube-api-access-nbcr6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.114001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.113945 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7ca03e-63bc-4107-85aa-73852f18cf40-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.114001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.113957 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd447616-cb5a-417a-994f-25acea127f8f-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:04.125924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.125887 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd10d8d-70a5-4897-81c8-0875ff56a289" path="/var/lib/kubelet/pods/abd10d8d-70a5-4897-81c8-0875ff56a289/volumes" Oct 30 06:19:04.202535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.202495 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh"] Oct 30 06:19:04.211397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.209748 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-p98mh"] Oct 30 06:19:04.481448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.481370 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:19:04.482589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.482567 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:19:04.815170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.815070 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" event={"ID":"fd447616-cb5a-417a-994f-25acea127f8f","Type":"ContainerDied","Data":"b2bd1a338f555e17b634e57f6dc0bfdfd054a8cf329a861492feeb64ece47edf"} Oct 30 06:19:04.815170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.815139 5001 scope.go:122] "RemoveContainer" containerID="65347044dfc1dcf85109df612336f773f80e88e56c168b77da2f49ca52d7f754" Oct 30 06:19:04.815170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.815089 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf" Oct 30 06:19:04.818552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.817965 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" Oct 30 06:19:04.818552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.818028 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl" event={"ID":"1c7ca03e-63bc-4107-85aa-73852f18cf40","Type":"ContainerDied","Data":"c9942f2bcacdbd3fc214c1b16f177f57a3b6f2686d9d359ae1ecbdb1f58a5565"} Oct 30 06:19:04.827545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.827523 5001 scope.go:122] "RemoveContainer" containerID="2ed43fa7a8dc76c8ba309717ded505c1ba20653e60821f6ffffcecd849e6f6c8" Oct 30 06:19:04.881344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.874831 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl"] Oct 30 06:19:04.890889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.890844 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-zpvwl"] Oct 30 06:19:04.924993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.924850 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf"] Oct 30 06:19:04.935048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:04.935013 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-wk7pf"] Oct 30 06:19:05.136486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:05.136448 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:19:05.136693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:05.136541 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:19:05.906976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:05.906918 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" podUID="86ebf91c-2596-4e19-93cb-566052ea639a" containerName="payload-processing" containerID="cri-o://9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e" gracePeriod=25 Oct 30 06:19:05.917607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:05.917542 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" podUID="dae76461-bda0-4a23-bcf6-e74c3a4ee951" containerName="payload-pre-processing" containerID="cri-o://f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d" gracePeriod=25 Oct 30 06:19:06.123376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.122210 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7ca03e-63bc-4107-85aa-73852f18cf40" path="/var/lib/kubelet/pods/1c7ca03e-63bc-4107-85aa-73852f18cf40/volumes" Oct 30 06:19:06.123376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.122767 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" path="/var/lib/kubelet/pods/69af5daa-6e0f-416c-b808-8a2ef5eaa603/volumes" Oct 30 06:19:06.123376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.123165 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd447616-cb5a-417a-994f-25acea127f8f" path="/var/lib/kubelet/pods/fd447616-cb5a-417a-994f-25acea127f8f/volumes" Oct 30 06:19:06.345538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.343497 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:19:06.414364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.412400 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:19:06.495966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.495870 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae76461-bda0-4a23-bcf6-e74c3a4ee951-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae76461-bda0-4a23-bcf6-e74c3a4ee951-plugins-config-volume\") pod \"dae76461-bda0-4a23-bcf6-e74c3a4ee951\" (UID: \"dae76461-bda0-4a23-bcf6-e74c3a4ee951\") " Oct 30 06:19:06.496279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.496256 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ebf91c-2596-4e19-93cb-566052ea639a-kube-api-access-dffwc\" (UniqueName: \"kubernetes.io/projected/86ebf91c-2596-4e19-93cb-566052ea639a-kube-api-access-dffwc\") pod \"86ebf91c-2596-4e19-93cb-566052ea639a\" (UID: \"86ebf91c-2596-4e19-93cb-566052ea639a\") " Oct 30 06:19:06.496510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.496493 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae76461-bda0-4a23-bcf6-e74c3a4ee951-kube-api-access-nsktp\" (UniqueName: \"kubernetes.io/projected/dae76461-bda0-4a23-bcf6-e74c3a4ee951-kube-api-access-nsktp\") pod \"dae76461-bda0-4a23-bcf6-e74c3a4ee951\" (UID: \"dae76461-bda0-4a23-bcf6-e74c3a4ee951\") " Oct 30 06:19:06.496727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.496711 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ebf91c-2596-4e19-93cb-566052ea639a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ebf91c-2596-4e19-93cb-566052ea639a-plugins-config-volume\") pod \"86ebf91c-2596-4e19-93cb-566052ea639a\" (UID: \"86ebf91c-2596-4e19-93cb-566052ea639a\") " Oct 30 06:19:06.497995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.497186 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ebf91c-2596-4e19-93cb-566052ea639a-plugins-config-volume" pod "86ebf91c-2596-4e19-93cb-566052ea639a" (UID: "86ebf91c-2596-4e19-93cb-566052ea639a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:06.497995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.497673 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae76461-bda0-4a23-bcf6-e74c3a4ee951-plugins-config-volume" pod "dae76461-bda0-4a23-bcf6-e74c3a4ee951" (UID: "dae76461-bda0-4a23-bcf6-e74c3a4ee951"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:06.499119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.499094 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae76461-bda0-4a23-bcf6-e74c3a4ee951-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:06.499243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.499231 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ebf91c-2596-4e19-93cb-566052ea639a-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:06.501518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.501473 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ebf91c-2596-4e19-93cb-566052ea639a-kube-api-access-dffwc" pod "86ebf91c-2596-4e19-93cb-566052ea639a" (UID: "86ebf91c-2596-4e19-93cb-566052ea639a"). InnerVolumeSpecName "kube-api-access-dffwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:06.503009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.502986 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae76461-bda0-4a23-bcf6-e74c3a4ee951-kube-api-access-nsktp" pod "dae76461-bda0-4a23-bcf6-e74c3a4ee951" (UID: "dae76461-bda0-4a23-bcf6-e74c3a4ee951"). InnerVolumeSpecName "kube-api-access-nsktp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:06.600635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.600598 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dffwc\" (UniqueName: \"kubernetes.io/projected/86ebf91c-2596-4e19-93cb-566052ea639a-kube-api-access-dffwc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:06.600635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.600631 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsktp\" (UniqueName: \"kubernetes.io/projected/dae76461-bda0-4a23-bcf6-e74c3a4ee951-kube-api-access-nsktp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:06.842825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.842733 5001 generic.go:356] "Generic (PLEG): container finished" podID="dae76461-bda0-4a23-bcf6-e74c3a4ee951" containerID="f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d" exitCode=0 Oct 30 06:19:06.842825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.842802 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" Oct 30 06:19:06.843080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.842820 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" event={"ID":"dae76461-bda0-4a23-bcf6-e74c3a4ee951","Type":"ContainerDied","Data":"f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d"} Oct 30 06:19:06.843080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.842906 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm" event={"ID":"dae76461-bda0-4a23-bcf6-e74c3a4ee951","Type":"ContainerDied","Data":"95e4bf4a9a50c8623b9c82f859e9dc7ff7962dcced44be1e72590dc69b27d22b"} Oct 30 06:19:06.843080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.842939 5001 scope.go:122] "RemoveContainer" containerID="f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d" Oct 30 06:19:06.844862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.844836 5001 generic.go:356] "Generic (PLEG): container finished" podID="86ebf91c-2596-4e19-93cb-566052ea639a" containerID="9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e" exitCode=0 Oct 30 06:19:06.845140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.844919 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" Oct 30 06:19:06.845140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.844963 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" event={"ID":"86ebf91c-2596-4e19-93cb-566052ea639a","Type":"ContainerDied","Data":"9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e"} Oct 30 06:19:06.845140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.845015 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx" event={"ID":"86ebf91c-2596-4e19-93cb-566052ea639a","Type":"ContainerDied","Data":"eace102f7af62019ed2eb4c3875afd4a9001f45992266f1d81237a9adab71e0e"} Oct 30 06:19:06.873547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.872354 5001 scope.go:122] "RemoveContainer" containerID="f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d" Oct 30 06:19:06.887335 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:19:06.878469 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d\": container with ID starting with f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d not found: ID does not exist" containerID="f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d" Oct 30 06:19:06.887335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.878516 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d"} err="failed to get container status \"f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d\": rpc error: code = NotFound desc = could not find container \"f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d\": container with ID starting with f5d4c765d0792ce7f3a6f63171c2c0468b24d93d33f665eed4662bd214b7f34d not found: ID does not exist" Oct 30 06:19:06.887335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.878544 5001 scope.go:122] "RemoveContainer" containerID="9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e" Oct 30 06:19:06.901055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.901029 5001 scope.go:122] "RemoveContainer" containerID="9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e" Oct 30 06:19:06.901809 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:19:06.901785 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e\": container with ID starting with 9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e not found: ID does not exist" containerID="9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e" Oct 30 06:19:06.901950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.901930 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e"} err="failed to get container status \"9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e\": rpc error: code = NotFound desc = could not find container \"9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e\": container with ID starting with 9a0cb4cbbb7b775bd49ab2ef9d71fc94112374b867ac0d84b93422574f62bd9e not found: ID does not exist" Oct 30 06:19:06.916237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.916204 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx"] Oct 30 06:19:06.920415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.920388 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-665d74b76b-6rnkx"] Oct 30 06:19:06.989687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.989636 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm"] Oct 30 06:19:06.994855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:06.994816 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-6479dd9979-tw8gm"] Oct 30 06:19:07.705333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:07.703122 5001 scope.go:122] "RemoveContainer" containerID="fd23e40793b224b87684f205673c0d20ae02e86104eef6a9e03be7966429c8a2" Oct 30 06:19:07.713739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:07.713720 5001 scope.go:122] "RemoveContainer" containerID="8ea22c447766dc60b288fae213ecd351082f825a50ccc97ef7739210d2fd75ef" Oct 30 06:19:08.118294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:08.118261 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ebf91c-2596-4e19-93cb-566052ea639a" path="/var/lib/kubelet/pods/86ebf91c-2596-4e19-93cb-566052ea639a/volumes" Oct 30 06:19:08.118712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:08.118698 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae76461-bda0-4a23-bcf6-e74c3a4ee951" path="/var/lib/kubelet/pods/dae76461-bda0-4a23-bcf6-e74c3a4ee951/volumes" Oct 30 06:19:08.302835 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:19:08.302800 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:19:10.902158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.902117 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5"] Oct 30 06:19:10.903101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903080 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f3b400d0-7c38-4d95-8a56-7fdd98a39f5a" containerName="payload-pre-processing" Oct 30 06:19:10.903149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903105 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b400d0-7c38-4d95-8a56-7fdd98a39f5a" containerName="payload-pre-processing" Oct 30 06:19:10.903149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903120 5001 cpu_manager.go:414] "containerMap: removing container" podUID="58b8efdd-c28a-4818-a8a5-fbd3a422669e" containerName="payload-processing" Oct 30 06:19:10.903149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903129 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b8efdd-c28a-4818-a8a5-fbd3a422669e" containerName="payload-processing" Oct 30 06:19:10.903149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903145 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dae76461-bda0-4a23-bcf6-e74c3a4ee951" containerName="payload-pre-processing" Oct 30 06:19:10.903277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903154 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae76461-bda0-4a23-bcf6-e74c3a4ee951" containerName="payload-pre-processing" Oct 30 06:19:10.903277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903165 5001 cpu_manager.go:414] "containerMap: removing container" podUID="86ebf91c-2596-4e19-93cb-566052ea639a" containerName="payload-processing" Oct 30 06:19:10.903277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903172 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ebf91c-2596-4e19-93cb-566052ea639a" containerName="payload-processing" Oct 30 06:19:10.903277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903183 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ddcce932-f11c-45af-b2a8-3563da5b90ac" containerName="payload-pre-processing" Oct 30 06:19:10.903277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903190 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddcce932-f11c-45af-b2a8-3563da5b90ac" containerName="payload-pre-processing" Oct 30 06:19:10.903277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903215 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1c7ca03e-63bc-4107-85aa-73852f18cf40" containerName="istio-proxy" Oct 30 06:19:10.903277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903222 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7ca03e-63bc-4107-85aa-73852f18cf40" containerName="istio-proxy" Oct 30 06:19:10.903277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903245 5001 cpu_manager.go:414] "containerMap: removing container" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" Oct 30 06:19:10.903277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903252 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903293 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a5bda487-9e0e-40da-8340-7577ab18ca42" containerName="payload-processing" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903300 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bda487-9e0e-40da-8340-7577ab18ca42" containerName="payload-processing" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903331 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fd447616-cb5a-417a-994f-25acea127f8f" containerName="istio-proxy" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903338 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd447616-cb5a-417a-994f-25acea127f8f" containerName="istio-proxy" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903349 5001 cpu_manager.go:414] "containerMap: removing container" podUID="abd10d8d-70a5-4897-81c8-0875ff56a289" containerName="istio-proxy" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903356 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd10d8d-70a5-4897-81c8-0875ff56a289" containerName="istio-proxy" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903520 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b400d0-7c38-4d95-8a56-7fdd98a39f5a" containerName="payload-pre-processing" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903530 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b8efdd-c28a-4818-a8a5-fbd3a422669e" containerName="payload-processing" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903541 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae76461-bda0-4a23-bcf6-e74c3a4ee951" containerName="payload-pre-processing" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903548 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ebf91c-2596-4e19-93cb-566052ea639a" containerName="payload-processing" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903556 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddcce932-f11c-45af-b2a8-3563da5b90ac" containerName="payload-pre-processing" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903566 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7ca03e-63bc-4107-85aa-73852f18cf40" containerName="istio-proxy" Oct 30 06:19:10.903577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903576 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="69af5daa-6e0f-416c-b808-8a2ef5eaa603" containerName="istio-proxy" Oct 30 06:19:10.903970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903593 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bda487-9e0e-40da-8340-7577ab18ca42" containerName="payload-processing" Oct 30 06:19:10.903970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903602 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd447616-cb5a-417a-994f-25acea127f8f" containerName="istio-proxy" Oct 30 06:19:10.903970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.903609 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd10d8d-70a5-4897-81c8-0875ff56a289" containerName="istio-proxy" Oct 30 06:19:10.906512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.906490 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:19:10.918407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.918377 5001 generic.go:356] "Generic (PLEG): container finished" podID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerID="4a2e64bafe5b33136118b72d8e7eff8112424a287d34e962d7dc85a13908b56b" exitCode=0 Oct 30 06:19:10.918554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.918469 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" event={"ID":"82997d1a-74de-4f2f-b8cb-47aa1fab159c","Type":"ContainerDied","Data":"4a2e64bafe5b33136118b72d8e7eff8112424a287d34e962d7dc85a13908b56b"} Oct 30 06:19:10.919025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.919003 5001 scope.go:122] "RemoveContainer" containerID="4a2e64bafe5b33136118b72d8e7eff8112424a287d34e962d7dc85a13908b56b" Oct 30 06:19:10.923373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.923344 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:10.923496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.923412 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:10.923496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.923449 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:10.924483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.924453 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ddedc39-8fc0-4569-b483-6af92b36eacb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5\" (UID: \"1ddedc39-8fc0-4569-b483-6af92b36eacb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:19:10.924602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.924585 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gcsn\" (UniqueName: \"kubernetes.io/projected/1ddedc39-8fc0-4569-b483-6af92b36eacb-kube-api-access-9gcsn\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5\" (UID: \"1ddedc39-8fc0-4569-b483-6af92b36eacb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:19:10.927105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.927083 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5"] Oct 30 06:19:10.931942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.931918 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-532f3b\"" Oct 30 06:19:10.932090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:10.932074 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-532f3b-dockercfg-dqjvp\"" Oct 30 06:19:11.158386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.152243 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ddedc39-8fc0-4569-b483-6af92b36eacb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5\" (UID: \"1ddedc39-8fc0-4569-b483-6af92b36eacb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:19:11.158386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.152461 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gcsn\" (UniqueName: \"kubernetes.io/projected/1ddedc39-8fc0-4569-b483-6af92b36eacb-kube-api-access-9gcsn\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5\" (UID: \"1ddedc39-8fc0-4569-b483-6af92b36eacb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:19:11.158386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.153960 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ddedc39-8fc0-4569-b483-6af92b36eacb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5\" (UID: \"1ddedc39-8fc0-4569-b483-6af92b36eacb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:19:11.204327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.198919 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2"] Oct 30 06:19:11.204327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.203767 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:19:11.227961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.227923 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2"] Oct 30 06:19:11.291344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.270609 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d923d506-2ebd-470d-b04d-ffa47379c56a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2\" (UID: \"d923d506-2ebd-470d-b04d-ffa47379c56a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:19:11.291344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.270798 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4csp7\" (UniqueName: \"kubernetes.io/projected/d923d506-2ebd-470d-b04d-ffa47379c56a-kube-api-access-4csp7\") pod \"payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2\" (UID: \"d923d506-2ebd-470d-b04d-ffa47379c56a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:19:11.313360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.311823 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gcsn\" (UniqueName: \"kubernetes.io/projected/1ddedc39-8fc0-4569-b483-6af92b36eacb-kube-api-access-9gcsn\") pod \"payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5\" (UID: \"1ddedc39-8fc0-4569-b483-6af92b36eacb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:19:11.385332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.384991 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d923d506-2ebd-470d-b04d-ffa47379c56a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2\" (UID: \"d923d506-2ebd-470d-b04d-ffa47379c56a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:19:11.385332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.385238 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4csp7\" (UniqueName: \"kubernetes.io/projected/d923d506-2ebd-470d-b04d-ffa47379c56a-kube-api-access-4csp7\") pod \"payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2\" (UID: \"d923d506-2ebd-470d-b04d-ffa47379c56a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:19:11.386657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.386624 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d923d506-2ebd-470d-b04d-ffa47379c56a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2\" (UID: \"d923d506-2ebd-470d-b04d-ffa47379c56a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:19:11.432706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.432524 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4csp7\" (UniqueName: \"kubernetes.io/projected/d923d506-2ebd-470d-b04d-ffa47379c56a-kube-api-access-4csp7\") pod \"payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2\" (UID: \"d923d506-2ebd-470d-b04d-ffa47379c56a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:19:11.526070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.526024 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:19:11.636140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.636103 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:19:11.887201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.887156 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5"] Oct 30 06:19:11.914091 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:19:11.914062 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddedc39_8fc0_4569_b483_6af92b36eacb.slice/crio-377c33ffb00a864b38c1f23c6e47e73a1256c23a06079ad1aa3b75fc07dd0092 WatchSource:0}: Error finding container 377c33ffb00a864b38c1f23c6e47e73a1256c23a06079ad1aa3b75fc07dd0092: Status 404 returned error can't find the container with id 377c33ffb00a864b38c1f23c6e47e73a1256c23a06079ad1aa3b75fc07dd0092 Oct 30 06:19:11.958511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.958402 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" event={"ID":"1ddedc39-8fc0-4569-b483-6af92b36eacb","Type":"ContainerStarted","Data":"377c33ffb00a864b38c1f23c6e47e73a1256c23a06079ad1aa3b75fc07dd0092"} Oct 30 06:19:11.961695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:11.961651 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" event={"ID":"82997d1a-74de-4f2f-b8cb-47aa1fab159c","Type":"ContainerStarted","Data":"fb9cc9241688c255e9cbb8ac6c0778f463d76e76adbbc82977642a2b6d76f651"} Oct 30 06:19:12.024383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:12.024352 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2"] Oct 30 06:19:12.027792 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:19:12.027759 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd923d506_2ebd_470d_b04d_ffa47379c56a.slice/crio-9e3219607f3e0ec091b0e935b67f91960dd242ad38cb94412732fa2a4cf967c2 WatchSource:0}: Error finding container 9e3219607f3e0ec091b0e935b67f91960dd242ad38cb94412732fa2a4cf967c2: Status 404 returned error can't find the container with id 9e3219607f3e0ec091b0e935b67f91960dd242ad38cb94412732fa2a4cf967c2 Oct 30 06:19:12.123934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:12.123889 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:19:12.124447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:12.124405 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 06:19:12.141409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:12.141374 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:19:13.011329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.010958 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" event={"ID":"1ddedc39-8fc0-4569-b483-6af92b36eacb","Type":"ContainerStarted","Data":"324ae686b42bdc043a74cbcdb3ade810c1c308c04ca351e3e5c5cef2b8b56f8b"} Oct 30 06:19:13.040331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.032121 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:19:13.079920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.076380 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv"] Oct 30 06:19:13.107276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.107125 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" event={"ID":"d923d506-2ebd-470d-b04d-ffa47379c56a","Type":"ContainerStarted","Data":"6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007"} Oct 30 06:19:13.107276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.107180 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" event={"ID":"d923d506-2ebd-470d-b04d-ffa47379c56a","Type":"ContainerStarted","Data":"9e3219607f3e0ec091b0e935b67f91960dd242ad38cb94412732fa2a4cf967c2"} Oct 30 06:19:13.107276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.107235 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:19:13.137331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.133941 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:19:13.137331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.134002 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 06:19:13.180744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.177758 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3267419-tmkrv"] Oct 30 06:19:13.200066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.193862 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" podStartSLOduration=3.193838693 podStartE2EDuration="3.193838693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:13.102432605 +0000 UTC m=+1567.098389616" watchObservedRunningTime="2026-10-30 06:19:13.193838693 +0000 UTC m=+1567.189795707" Oct 30 06:19:13.225694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.225602 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" podStartSLOduration=2.225577512 podStartE2EDuration="2.225577512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:13.167000252 +0000 UTC m=+1567.162957263" watchObservedRunningTime="2026-10-30 06:19:13.225577512 +0000 UTC m=+1567.221534524" Oct 30 06:19:13.634112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:13.634073 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:19:14.120722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:14.120676 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e3754b-41d7-468f-bd6a-153bd16045a8" path="/var/lib/kubelet/pods/89e3754b-41d7-468f-bd6a-153bd16045a8/volumes" Oct 30 06:19:14.121388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:14.121365 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:19:14.121509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:14.121416 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 06:19:15.121587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:15.121552 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:19:15.122045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:15.121609 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 06:19:15.123935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:15.123906 5001 generic.go:356] "Generic (PLEG): container finished" podID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerID="e5f44e7b4363c1a0ad75843d9feb6b8c070f25f0212576bfa3681c2b90f5100f" exitCode=0 Oct 30 06:19:15.124082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:15.123967 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" event={"ID":"02cdba52-fb35-42fa-91f2-77ecec35b5ef","Type":"ContainerDied","Data":"e5f44e7b4363c1a0ad75843d9feb6b8c070f25f0212576bfa3681c2b90f5100f"} Oct 30 06:19:15.124672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:15.124652 5001 scope.go:122] "RemoveContainer" containerID="e5f44e7b4363c1a0ad75843d9feb6b8c070f25f0212576bfa3681c2b90f5100f" Oct 30 06:19:15.126072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:15.126043 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:15.126124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:15.126112 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:15.126159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:15.126146 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:16.121884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:16.121853 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:19:16.122333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:16.121902 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 06:19:16.134663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:16.134630 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" event={"ID":"02cdba52-fb35-42fa-91f2-77ecec35b5ef","Type":"ContainerStarted","Data":"da48dd99c1ee6e995240909fd08725313186c9afdd385858b5df069b79f2eb6a"} Oct 30 06:19:16.514411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:16.514321 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:19:16.515341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:16.515317 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 06:19:16.515466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:16.515361 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 06:19:17.121531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:17.121492 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:19:17.121747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:17.121553 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 06:19:17.513958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:17.513859 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 06:19:17.513958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:17.513917 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 06:19:18.139513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:18.139479 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:19:18.139979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:18.139952 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:19:18.141171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:18.141149 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:19:18.514213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:18.514125 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 06:19:18.514213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:18.514187 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 06:19:19.514451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:19.514419 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 06:19:19.514876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:19.514472 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 06:19:20.104790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:20.104756 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:19:20.513882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:20.513796 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 06:19:20.513882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:20.513846 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 06:19:21.514605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:21.514568 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 06:19:21.515033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:21.514628 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 06:19:22.519034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:22.519004 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:19:22.520125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:22.520104 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:19:23.048173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:23.048132 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx"] Oct 30 06:19:23.051917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:23.051881 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4959ac-4tcpx"] Oct 30 06:19:24.118801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:24.118765 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ad37e2-90ac-4105-8175-4245b01a3bf7" path="/var/lib/kubelet/pods/95ad37e2-90ac-4105-8175-4245b01a3bf7/volumes" Oct 30 06:19:24.119201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:24.119132 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:19:25.126754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:25.126721 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:19:27.186354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.186279 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9"] Oct 30 06:19:27.191992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.191965 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:19:27.213291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.213256 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9"] Oct 30 06:19:27.218974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.218946 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a55a4b\"" Oct 30 06:19:27.219186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.219021 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a55a4b-dockercfg-2m2m2\"" Oct 30 06:19:27.376332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.322348 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmks8\" (UniqueName: \"kubernetes.io/projected/5e5a1389-012c-4d7e-968e-b51a8bb781d6-kube-api-access-rmks8\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9\" (UID: \"5e5a1389-012c-4d7e-968e-b51a8bb781d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:19:27.376976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.376941 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5a1389-012c-4d7e-968e-b51a8bb781d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9\" (UID: \"5e5a1389-012c-4d7e-968e-b51a8bb781d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:19:27.490395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.484378 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmks8\" (UniqueName: \"kubernetes.io/projected/5e5a1389-012c-4d7e-968e-b51a8bb781d6-kube-api-access-rmks8\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9\" (UID: \"5e5a1389-012c-4d7e-968e-b51a8bb781d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:19:27.490395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.484718 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5a1389-012c-4d7e-968e-b51a8bb781d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9\" (UID: \"5e5a1389-012c-4d7e-968e-b51a8bb781d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:19:27.499274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.499235 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5a1389-012c-4d7e-968e-b51a8bb781d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9\" (UID: \"5e5a1389-012c-4d7e-968e-b51a8bb781d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:19:27.505488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.503370 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l"] Oct 30 06:19:27.537392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.537361 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l"] Oct 30 06:19:27.537766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.537746 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:19:27.540044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.540016 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmks8\" (UniqueName: \"kubernetes.io/projected/5e5a1389-012c-4d7e-968e-b51a8bb781d6-kube-api-access-rmks8\") pod \"payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9\" (UID: \"5e5a1389-012c-4d7e-968e-b51a8bb781d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:19:27.687964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.687918 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l\" (UID: \"af3b8f19-f99e-4299-a4bf-bd9cbabafc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:19:27.688197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.688171 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glffx\" (UniqueName: \"kubernetes.io/projected/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-kube-api-access-glffx\") pod \"payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l\" (UID: \"af3b8f19-f99e-4299-a4bf-bd9cbabafc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:19:27.789228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.789135 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glffx\" (UniqueName: \"kubernetes.io/projected/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-kube-api-access-glffx\") pod \"payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l\" (UID: \"af3b8f19-f99e-4299-a4bf-bd9cbabafc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:19:27.789418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.789292 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l\" (UID: \"af3b8f19-f99e-4299-a4bf-bd9cbabafc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:19:27.790218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.790189 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l\" (UID: \"af3b8f19-f99e-4299-a4bf-bd9cbabafc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:19:27.811915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.811874 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:19:27.812168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.812148 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glffx\" (UniqueName: \"kubernetes.io/projected/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-kube-api-access-glffx\") pod \"payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l\" (UID: \"af3b8f19-f99e-4299-a4bf-bd9cbabafc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:19:27.906329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:27.901018 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:19:28.156640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:28.156599 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9"] Oct 30 06:19:28.163728 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:19:28.163684 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5a1389_012c_4d7e_968e_b51a8bb781d6.slice/crio-52e8011513290362a68a8dfe58391e3ec844e06ab49dc2d0a51f72421561d836 WatchSource:0}: Error finding container 52e8011513290362a68a8dfe58391e3ec844e06ab49dc2d0a51f72421561d836: Status 404 returned error can't find the container with id 52e8011513290362a68a8dfe58391e3ec844e06ab49dc2d0a51f72421561d836 Oct 30 06:19:28.260953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:28.260921 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" event={"ID":"5e5a1389-012c-4d7e-968e-b51a8bb781d6","Type":"ContainerStarted","Data":"52e8011513290362a68a8dfe58391e3ec844e06ab49dc2d0a51f72421561d836"} Oct 30 06:19:28.295237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:28.295211 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l"] Oct 30 06:19:28.297737 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:19:28.297705 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3b8f19_f99e_4299_a4bf_bd9cbabafc49.slice/crio-1c0c9621223ac861061a4a31710d376fe696936c097f4fd569a3e70353d51617 WatchSource:0}: Error finding container 1c0c9621223ac861061a4a31710d376fe696936c097f4fd569a3e70353d51617: Status 404 returned error can't find the container with id 1c0c9621223ac861061a4a31710d376fe696936c097f4fd569a3e70353d51617 Oct 30 06:19:28.727321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:28.727022 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5"] Oct 30 06:19:28.738240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:28.731608 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:28.753369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:28.748841 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f2cf8a-openshift-default-dockercfg-pvtwv\"" Oct 30 06:19:28.795017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:28.794036 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5"] Oct 30 06:19:29.173744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.173710 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-credential-socket\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.174046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.174010 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-token\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.174481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.174450 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-certs\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.175223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.175200 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-podinfo\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.175435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.175403 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9dhc\" (UniqueName: \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-kube-api-access-g9dhc\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.183325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.175985 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istiod-ca-cert\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.184386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.184356 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-data\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.195888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.195851 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-envoy\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.197975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.197767 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-socket\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.286329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.282776 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" event={"ID":"af3b8f19-f99e-4299-a4bf-bd9cbabafc49","Type":"ContainerStarted","Data":"a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846"} Oct 30 06:19:29.286329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.282843 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" event={"ID":"af3b8f19-f99e-4299-a4bf-bd9cbabafc49","Type":"ContainerStarted","Data":"1c0c9621223ac861061a4a31710d376fe696936c097f4fd569a3e70353d51617"} Oct 30 06:19:29.286329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.283918 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:19:29.301326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.294470 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" event={"ID":"5e5a1389-012c-4d7e-968e-b51a8bb781d6","Type":"ContainerStarted","Data":"af91c76d7597b8ebf3303e37f1617a1827e7a5666cffd5a8e02c87fec30961b3"} Oct 30 06:19:29.301326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.295010 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:19:29.321380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.321343 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-envoy\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.321521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.321406 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-socket\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.321584 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.321554 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-credential-socket\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.321639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.321582 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-token\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.321758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.321735 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-certs\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.321886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.321844 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-podinfo\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.322021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.321913 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9dhc\" (UniqueName: \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-kube-api-access-g9dhc\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.322021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.321968 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istiod-ca-cert\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.322157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.322099 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-credential-socket\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.322157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.322112 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-data\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.322617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.322594 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-data\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.322716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.322622 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-certs\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.322844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.322825 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-socket\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.322991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.322971 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istiod-ca-cert\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.324724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.324697 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-envoy\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.324987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.324971 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-podinfo\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.333545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.333500 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" podStartSLOduration=2.333488456 podStartE2EDuration="2.333488456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:29.33088003 +0000 UTC m=+1583.326837051" watchObservedRunningTime="2026-10-30 06:19:29.333488456 +0000 UTC m=+1583.329445465" Oct 30 06:19:29.347503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.347477 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-token\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.348404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.348384 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9dhc\" (UniqueName: \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-kube-api-access-g9dhc\") pod \"e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.349240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.349219 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:29.386808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.384820 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" podStartSLOduration=2.384798827 podStartE2EDuration="2.384798827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:29.380760537 +0000 UTC m=+1583.376717560" watchObservedRunningTime="2026-10-30 06:19:29.384798827 +0000 UTC m=+1583.380755840" Oct 30 06:19:29.822832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.820939 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5"] Oct 30 06:19:29.835688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.835026 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:29.835688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.835118 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:29.835688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:29.835188 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:30.306639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:30.306599 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" event={"ID":"10785405-d520-4f7f-9ddc-af67c7c5a1bf","Type":"ContainerStarted","Data":"7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e"} Oct 30 06:19:30.307073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:30.306661 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" event={"ID":"10785405-d520-4f7f-9ddc-af67c7c5a1bf","Type":"ContainerStarted","Data":"96d4c432540d8d26ab53465f8b7eebefe7465ce957bec2ad8d64b58f5e2adbe9"} Oct 30 06:19:30.350046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:30.350007 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:30.369861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:30.369801 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" podStartSLOduration=2.369784943 podStartE2EDuration="2.369784943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:30.36355757 +0000 UTC m=+1584.359514581" watchObservedRunningTime="2026-10-30 06:19:30.369784943 +0000 UTC m=+1584.365741952" Oct 30 06:19:31.381733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:31.381704 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:31.863339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:31.859622 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97"] Oct 30 06:19:31.934536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:31.924158 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:19:31.963330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:31.958672 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8272af\"" Oct 30 06:19:31.963330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:31.959670 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8272af-dockercfg-qmvmz\"" Oct 30 06:19:32.002624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:31.992418 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97"] Oct 30 06:19:32.045786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.045653 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl"] Oct 30 06:19:32.075779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.075748 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:19:32.083340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.079035 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97\" (UID: \"7bc81a5c-ef66-4f6d-80f5-65abb5264d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:19:32.083340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.079359 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hglk\" (UniqueName: \"kubernetes.io/projected/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-kube-api-access-7hglk\") pod \"payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97\" (UID: \"7bc81a5c-ef66-4f6d-80f5-65abb5264d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:19:32.099171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.099132 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl"] Oct 30 06:19:32.188475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.183551 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hglk\" (UniqueName: \"kubernetes.io/projected/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-kube-api-access-7hglk\") pod \"payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97\" (UID: \"7bc81a5c-ef66-4f6d-80f5-65abb5264d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:19:32.188475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.183651 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d996\" (UniqueName: \"kubernetes.io/projected/2dfaafe5-2935-4184-8c71-78644c4f7d5d-kube-api-access-7d996\") pod \"payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl\" (UID: \"2dfaafe5-2935-4184-8c71-78644c4f7d5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:19:32.188475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.183710 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97\" (UID: \"7bc81a5c-ef66-4f6d-80f5-65abb5264d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:19:32.188475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.183915 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfaafe5-2935-4184-8c71-78644c4f7d5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl\" (UID: \"2dfaafe5-2935-4184-8c71-78644c4f7d5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:19:32.188475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.185255 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97\" (UID: \"7bc81a5c-ef66-4f6d-80f5-65abb5264d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:19:32.209117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.209078 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hglk\" (UniqueName: \"kubernetes.io/projected/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-kube-api-access-7hglk\") pod \"payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97\" (UID: \"7bc81a5c-ef66-4f6d-80f5-65abb5264d82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:19:32.285132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.285088 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfaafe5-2935-4184-8c71-78644c4f7d5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl\" (UID: \"2dfaafe5-2935-4184-8c71-78644c4f7d5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:19:32.285334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.285229 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d996\" (UniqueName: \"kubernetes.io/projected/2dfaafe5-2935-4184-8c71-78644c4f7d5d-kube-api-access-7d996\") pod \"payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl\" (UID: \"2dfaafe5-2935-4184-8c71-78644c4f7d5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:19:32.286070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.286045 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfaafe5-2935-4184-8c71-78644c4f7d5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl\" (UID: \"2dfaafe5-2935-4184-8c71-78644c4f7d5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:19:32.286260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.286244 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:19:32.308838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.308809 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d996\" (UniqueName: \"kubernetes.io/projected/2dfaafe5-2935-4184-8c71-78644c4f7d5d-kube-api-access-7d996\") pod \"payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl\" (UID: \"2dfaafe5-2935-4184-8c71-78644c4f7d5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:19:32.328534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.328174 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:32.329420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.329394 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:19:32.415358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.415328 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:19:32.623958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.623913 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97"] Oct 30 06:19:32.876973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:32.876867 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl"] Oct 30 06:19:33.340392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:33.340349 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" event={"ID":"7bc81a5c-ef66-4f6d-80f5-65abb5264d82","Type":"ContainerStarted","Data":"23fcfa351b696218279e552fd565d6592c137c35b7c6b35a172fc88370772ad6"} Oct 30 06:19:33.340626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:33.340421 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" event={"ID":"7bc81a5c-ef66-4f6d-80f5-65abb5264d82","Type":"ContainerStarted","Data":"6a2bf1d11ae6f8e81df1a8af3698a6ee36726e69edd548e2e94d532471f26787"} Oct 30 06:19:33.340626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:33.340449 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:19:33.343350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:33.343297 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" event={"ID":"2dfaafe5-2935-4184-8c71-78644c4f7d5d","Type":"ContainerStarted","Data":"4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b"} Oct 30 06:19:33.343478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:33.343365 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" event={"ID":"2dfaafe5-2935-4184-8c71-78644c4f7d5d","Type":"ContainerStarted","Data":"acac8653399751e602eaee95a103502a7fdcb5feca0db65288220661586459d7"} Oct 30 06:19:33.343550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:33.343481 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:19:33.391493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:33.391431 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" podStartSLOduration=2.391410842 podStartE2EDuration="2.391410842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:33.387544557 +0000 UTC m=+1587.383501568" watchObservedRunningTime="2026-10-30 06:19:33.391410842 +0000 UTC m=+1587.387367853" Oct 30 06:19:33.430878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:33.430824 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" podStartSLOduration=2.430809384 podStartE2EDuration="2.430809384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:33.427334048 +0000 UTC m=+1587.423291049" watchObservedRunningTime="2026-10-30 06:19:33.430809384 +0000 UTC m=+1587.426766447" Oct 30 06:19:35.010407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.010367 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w"] Oct 30 06:19:35.084671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.084635 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w"] Oct 30 06:19:35.085012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.084996 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:19:35.131207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.126619 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f2cf8a\"" Oct 30 06:19:35.131207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.126741 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f2cf8a-dockercfg-jqvjb\"" Oct 30 06:19:35.211488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.160967 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a23f0c-7108-43e4-aa56-06005da64513-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w\" (UID: \"88a23f0c-7108-43e4-aa56-06005da64513\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:19:35.211848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.211815 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxjg5\" (UniqueName: \"kubernetes.io/projected/88a23f0c-7108-43e4-aa56-06005da64513-kube-api-access-sxjg5\") pod \"payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w\" (UID: \"88a23f0c-7108-43e4-aa56-06005da64513\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:19:35.381331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.380239 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a23f0c-7108-43e4-aa56-06005da64513-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w\" (UID: \"88a23f0c-7108-43e4-aa56-06005da64513\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:19:35.381331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.380338 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxjg5\" (UniqueName: \"kubernetes.io/projected/88a23f0c-7108-43e4-aa56-06005da64513-kube-api-access-sxjg5\") pod \"payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w\" (UID: \"88a23f0c-7108-43e4-aa56-06005da64513\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:19:35.405856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.405786 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a23f0c-7108-43e4-aa56-06005da64513-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w\" (UID: \"88a23f0c-7108-43e4-aa56-06005da64513\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:19:35.413355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.413032 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxjg5\" (UniqueName: \"kubernetes.io/projected/88a23f0c-7108-43e4-aa56-06005da64513-kube-api-access-sxjg5\") pod \"payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w\" (UID: \"88a23f0c-7108-43e4-aa56-06005da64513\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:19:35.572052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.570586 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:19:35.895425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:35.895395 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w"] Oct 30 06:19:35.898268 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:19:35.898236 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a23f0c_7108_43e4_aa56_06005da64513.slice/crio-1dd8b4282fc3866db756efdaae64c5e22972170ef83a121a184630534e68fc09 WatchSource:0}: Error finding container 1dd8b4282fc3866db756efdaae64c5e22972170ef83a121a184630534e68fc09: Status 404 returned error can't find the container with id 1dd8b4282fc3866db756efdaae64c5e22972170ef83a121a184630534e68fc09 Oct 30 06:19:36.423395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:36.423354 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" event={"ID":"88a23f0c-7108-43e4-aa56-06005da64513","Type":"ContainerStarted","Data":"a0218856d4e45c043f86615c391a4b91763c02a4c568527f8c20c99b44eac5dd"} Oct 30 06:19:36.423957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:36.423419 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" event={"ID":"88a23f0c-7108-43e4-aa56-06005da64513","Type":"ContainerStarted","Data":"1dd8b4282fc3866db756efdaae64c5e22972170ef83a121a184630534e68fc09"} Oct 30 06:19:36.423957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:36.423476 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:19:36.482065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:36.482003 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" podStartSLOduration=2.48198702 podStartE2EDuration="2.48198702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:36.480677352 +0000 UTC m=+1590.476634363" watchObservedRunningTime="2026-10-30 06:19:36.48198702 +0000 UTC m=+1590.477944041" Oct 30 06:19:38.297322 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:19:38.291012 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:19:40.309493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:40.309464 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:19:41.321430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:41.321352 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:19:41.339884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:41.339855 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 06:19:41.340034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:41.339909 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 06:19:41.497583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:41.497495 5001 generic.go:356] "Generic (PLEG): container finished" podID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerID="82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d" exitCode=0 Oct 30 06:19:41.497769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:41.497701 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" event={"ID":"a63052f6-d65d-4307-93bb-daa7a5581e78","Type":"ContainerDied","Data":"82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d"} Oct 30 06:19:41.498183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:41.498161 5001 scope.go:122] "RemoveContainer" containerID="82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d" Oct 30 06:19:41.500015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:41.499986 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:41.500094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:41.500055 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:41.500094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:41.500087 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:42.605335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:42.605272 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" event={"ID":"a63052f6-d65d-4307-93bb-daa7a5581e78","Type":"ContainerStarted","Data":"c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006"} Oct 30 06:19:43.247732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:43.247694 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:19:43.247995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:43.247969 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 06:19:43.248091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:43.248043 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 06:19:44.247873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.247832 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 06:19:44.248295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.247912 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 06:19:44.305653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.305616 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c"] Oct 30 06:19:44.328564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.328533 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd"] Oct 30 06:19:44.369467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.369435 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:19:44.392353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.391473 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:19:44.913330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.912653 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j"] Oct 30 06:19:44.951669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.938572 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:44.980576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.968661 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j"] Oct 30 06:19:44.982984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.982062 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:19:44.982984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.982752 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:19:44.982984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.982852 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:19:44.982984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:44.982929 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:19:45.040438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.040396 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b129648a-3291-4760-aa07-57f3ec743256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:45.074715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.074677 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b129648a-3291-4760-aa07-57f3ec743256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:45.097331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.075051 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljxxm\" (UniqueName: \"kubernetes.io/projected/b129648a-3291-4760-aa07-57f3ec743256-kube-api-access-ljxxm\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:45.250974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.250876 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b129648a-3291-4760-aa07-57f3ec743256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:45.251540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.251134 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b129648a-3291-4760-aa07-57f3ec743256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:45.251540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.251175 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxxm\" (UniqueName: \"kubernetes.io/projected/b129648a-3291-4760-aa07-57f3ec743256-kube-api-access-ljxxm\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:45.314356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.251810 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b129648a-3291-4760-aa07-57f3ec743256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:45.314690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.288402 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 06:19:45.314903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.314852 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 06:19:45.339845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.339811 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b129648a-3291-4760-aa07-57f3ec743256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:45.340261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.340201 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxxm\" (UniqueName: \"kubernetes.io/projected/b129648a-3291-4760-aa07-57f3ec743256-kube-api-access-ljxxm\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:45.602816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.602714 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:45.941566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:45.941539 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j"] Oct 30 06:19:45.943610 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:19:45.943571 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb129648a_3291_4760_aa07_57f3ec743256.slice/crio-c3940e8fbfec27380016d506eafe46f03dcc2591547939fb955051cc208a9a9e WatchSource:0}: Error finding container c3940e8fbfec27380016d506eafe46f03dcc2591547939fb955051cc208a9a9e: Status 404 returned error can't find the container with id c3940e8fbfec27380016d506eafe46f03dcc2591547939fb955051cc208a9a9e Oct 30 06:19:46.184005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.183960 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:19:46.248162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.248062 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 06:19:46.248162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.248135 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 06:19:46.489812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.489789 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x"] Oct 30 06:19:46.497428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.497404 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x"] Oct 30 06:19:46.497527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.497501 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:46.602152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.602110 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f278\" (UniqueName: \"kubernetes.io/projected/76fef77f-35b2-4c64-a22d-293d4229df88-kube-api-access-4f278\") pod \"maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:46.602362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.602317 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:46.602415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.602365 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:46.646795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.646762 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" event={"ID":"b129648a-3291-4760-aa07-57f3ec743256","Type":"ContainerStarted","Data":"104f43372245e7c9331ba428e490609d4baae9b72e9b8f6fc6cb899867c65360"} Oct 30 06:19:46.647007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.646817 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" event={"ID":"b129648a-3291-4760-aa07-57f3ec743256","Type":"ContainerStarted","Data":"c3940e8fbfec27380016d506eafe46f03dcc2591547939fb955051cc208a9a9e"} Oct 30 06:19:46.691748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.691694 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" podStartSLOduration=2.6916776970000003 podStartE2EDuration="2.691677697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:46.687126074 +0000 UTC m=+1600.683083108" watchObservedRunningTime="2026-10-30 06:19:46.691677697 +0000 UTC m=+1600.687634707" Oct 30 06:19:46.703554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.703523 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f278\" (UniqueName: \"kubernetes.io/projected/76fef77f-35b2-4c64-a22d-293d4229df88-kube-api-access-4f278\") pod \"maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:46.703720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.703630 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:46.703720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.703671 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:46.704661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.704635 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:46.706237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.706216 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:46.726933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.726902 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f278\" (UniqueName: \"kubernetes.io/projected/76fef77f-35b2-4c64-a22d-293d4229df88-kube-api-access-4f278\") pod \"maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:46.822995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:46.822897 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:47.185219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:47.185179 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x"] Oct 30 06:19:47.227392 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:19:47.227354 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fef77f_35b2_4c64_a22d_293d4229df88.slice/crio-3b98600794df6384e1f6bf86249ddc3637fb71ac383fde7cd10f9c0fd7f8728e WatchSource:0}: Error finding container 3b98600794df6384e1f6bf86249ddc3637fb71ac383fde7cd10f9c0fd7f8728e: Status 404 returned error can't find the container with id 3b98600794df6384e1f6bf86249ddc3637fb71ac383fde7cd10f9c0fd7f8728e Oct 30 06:19:47.276472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:47.275491 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 06:19:47.276472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:47.275569 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 06:19:47.437820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:47.437736 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:19:47.723354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:47.723250 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" event={"ID":"76fef77f-35b2-4c64-a22d-293d4229df88","Type":"ContainerStarted","Data":"3b98600794df6384e1f6bf86249ddc3637fb71ac383fde7cd10f9c0fd7f8728e"} Oct 30 06:19:47.725855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:47.725826 5001 generic.go:356] "Generic (PLEG): container finished" podID="b129648a-3291-4760-aa07-57f3ec743256" containerID="104f43372245e7c9331ba428e490609d4baae9b72e9b8f6fc6cb899867c65360" exitCode=0 Oct 30 06:19:47.726222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:47.726200 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" event={"ID":"b129648a-3291-4760-aa07-57f3ec743256","Type":"ContainerDied","Data":"104f43372245e7c9331ba428e490609d4baae9b72e9b8f6fc6cb899867c65360"} Oct 30 06:19:48.433458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:48.433423 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:19:48.434721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:48.434694 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:19:48.435776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:48.435751 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:19:48.738649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:48.738557 5001 generic.go:356] "Generic (PLEG): container finished" podID="76fef77f-35b2-4c64-a22d-293d4229df88" containerID="5bdec4624a3b619ea4668eebc4168d755fa53be2e7c039d66850819fa17b8766" exitCode=0 Oct 30 06:19:48.739353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:48.738645 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" event={"ID":"76fef77f-35b2-4c64-a22d-293d4229df88","Type":"ContainerDied","Data":"5bdec4624a3b619ea4668eebc4168d755fa53be2e7c039d66850819fa17b8766"} Oct 30 06:19:48.932812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:48.932787 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:49.047769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.047678 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b129648a-3291-4760-aa07-57f3ec743256-kube-api-access-ljxxm\" (UniqueName: \"kubernetes.io/projected/b129648a-3291-4760-aa07-57f3ec743256-kube-api-access-ljxxm\") pod \"b129648a-3291-4760-aa07-57f3ec743256\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " Oct 30 06:19:49.047953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.047835 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b129648a-3291-4760-aa07-57f3ec743256-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b129648a-3291-4760-aa07-57f3ec743256-maas-api-openshift-ca\") pod \"b129648a-3291-4760-aa07-57f3ec743256\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " Oct 30 06:19:49.047953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.047916 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b129648a-3291-4760-aa07-57f3ec743256-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b129648a-3291-4760-aa07-57f3ec743256-maas-api-xks-ca\") pod \"b129648a-3291-4760-aa07-57f3ec743256\" (UID: \"b129648a-3291-4760-aa07-57f3ec743256\") " Oct 30 06:19:49.048816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.048459 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b129648a-3291-4760-aa07-57f3ec743256-maas-api-openshift-ca" pod "b129648a-3291-4760-aa07-57f3ec743256" (UID: "b129648a-3291-4760-aa07-57f3ec743256"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:49.048816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.048782 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b129648a-3291-4760-aa07-57f3ec743256-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:49.052256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.052195 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b129648a-3291-4760-aa07-57f3ec743256-kube-api-access-ljxxm" pod "b129648a-3291-4760-aa07-57f3ec743256" (UID: "b129648a-3291-4760-aa07-57f3ec743256"). InnerVolumeSpecName "kube-api-access-ljxxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:49.063749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.063702 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b129648a-3291-4760-aa07-57f3ec743256-maas-api-xks-ca" pod "b129648a-3291-4760-aa07-57f3ec743256" (UID: "b129648a-3291-4760-aa07-57f3ec743256"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:49.150623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.150590 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljxxm\" (UniqueName: \"kubernetes.io/projected/b129648a-3291-4760-aa07-57f3ec743256-kube-api-access-ljxxm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:49.150623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.150621 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b129648a-3291-4760-aa07-57f3ec743256-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:49.314578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.307380 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" podUID="aeb32039-e94a-43e6-83d2-054fcc21849e" containerName="payload-processing" containerID="cri-o://7685a8fdb4ecc0f530f0e390bd2a722a9d7e90de1a1899a0e6a1fefd01353f9a" gracePeriod=25 Oct 30 06:19:49.335128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.332387 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" podUID="1407ec9d-8fd0-4752-90ac-26b31e3d3035" containerName="payload-pre-processing" containerID="cri-o://356262877e35f0271de7fed51e17ca4cee24cf876a26c723d7838cb672f5aef9" gracePeriod=25 Oct 30 06:19:49.839296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.839245 5001 generic.go:356] "Generic (PLEG): container finished" podID="1407ec9d-8fd0-4752-90ac-26b31e3d3035" containerID="356262877e35f0271de7fed51e17ca4cee24cf876a26c723d7838cb672f5aef9" exitCode=0 Oct 30 06:19:49.839930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.839353 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" event={"ID":"1407ec9d-8fd0-4752-90ac-26b31e3d3035","Type":"ContainerDied","Data":"356262877e35f0271de7fed51e17ca4cee24cf876a26c723d7838cb672f5aef9"} Oct 30 06:19:49.844681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.844650 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" event={"ID":"b129648a-3291-4760-aa07-57f3ec743256","Type":"ContainerDied","Data":"c3940e8fbfec27380016d506eafe46f03dcc2591547939fb955051cc208a9a9e"} Oct 30 06:19:49.844920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.844905 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3940e8fbfec27380016d506eafe46f03dcc2591547939fb955051cc208a9a9e" Oct 30 06:19:49.845087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.845077 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j" Oct 30 06:19:49.851222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.851177 5001 generic.go:356] "Generic (PLEG): container finished" podID="aeb32039-e94a-43e6-83d2-054fcc21849e" containerID="7685a8fdb4ecc0f530f0e390bd2a722a9d7e90de1a1899a0e6a1fefd01353f9a" exitCode=0 Oct 30 06:19:49.851578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.851440 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" event={"ID":"aeb32039-e94a-43e6-83d2-054fcc21849e","Type":"ContainerDied","Data":"7685a8fdb4ecc0f530f0e390bd2a722a9d7e90de1a1899a0e6a1fefd01353f9a"} Oct 30 06:19:49.861609 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:19:49.861570 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1407ec9d_8fd0_4752_90ac_26b31e3d3035.slice/crio-conmon-356262877e35f0271de7fed51e17ca4cee24cf876a26c723d7838cb672f5aef9.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:19:49.886391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.886355 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq"] Oct 30 06:19:49.891477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:49.891447 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0fab6d-qcbpq"] Oct 30 06:19:50.124972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.123802 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:19:50.133437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.133402 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91db3de6-a8d4-407f-86b7-80fb8076186c" path="/var/lib/kubelet/pods/91db3de6-a8d4-407f-86b7-80fb8076186c/volumes" Oct 30 06:19:50.265769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.265736 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1407ec9d-8fd0-4752-90ac-26b31e3d3035-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1407ec9d-8fd0-4752-90ac-26b31e3d3035-plugins-config-volume\") pod \"1407ec9d-8fd0-4752-90ac-26b31e3d3035\" (UID: \"1407ec9d-8fd0-4752-90ac-26b31e3d3035\") " Oct 30 06:19:50.265913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.265839 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1407ec9d-8fd0-4752-90ac-26b31e3d3035-kube-api-access-w96ll\" (UniqueName: \"kubernetes.io/projected/1407ec9d-8fd0-4752-90ac-26b31e3d3035-kube-api-access-w96ll\") pod \"1407ec9d-8fd0-4752-90ac-26b31e3d3035\" (UID: \"1407ec9d-8fd0-4752-90ac-26b31e3d3035\") " Oct 30 06:19:50.271726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.267265 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1407ec9d-8fd0-4752-90ac-26b31e3d3035-plugins-config-volume" pod "1407ec9d-8fd0-4752-90ac-26b31e3d3035" (UID: "1407ec9d-8fd0-4752-90ac-26b31e3d3035"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:50.276247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.276208 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1407ec9d-8fd0-4752-90ac-26b31e3d3035-kube-api-access-w96ll" pod "1407ec9d-8fd0-4752-90ac-26b31e3d3035" (UID: "1407ec9d-8fd0-4752-90ac-26b31e3d3035"). InnerVolumeSpecName "kube-api-access-w96ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:50.284756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.284483 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:19:50.289543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.289521 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:50.369844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.369807 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1407ec9d-8fd0-4752-90ac-26b31e3d3035-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:50.369844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.369845 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w96ll\" (UniqueName: \"kubernetes.io/projected/1407ec9d-8fd0-4752-90ac-26b31e3d3035-kube-api-access-w96ll\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:50.470815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.470721 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb32039-e94a-43e6-83d2-054fcc21849e-kube-api-access-f7qbh\" (UniqueName: \"kubernetes.io/projected/aeb32039-e94a-43e6-83d2-054fcc21849e-kube-api-access-f7qbh\") pod \"aeb32039-e94a-43e6-83d2-054fcc21849e\" (UID: \"aeb32039-e94a-43e6-83d2-054fcc21849e\") " Oct 30 06:19:50.470815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.470777 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-xks-ca\") pod \"76fef77f-35b2-4c64-a22d-293d4229df88\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " Oct 30 06:19:50.471052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.470822 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb32039-e94a-43e6-83d2-054fcc21849e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb32039-e94a-43e6-83d2-054fcc21849e-plugins-config-volume\") pod \"aeb32039-e94a-43e6-83d2-054fcc21849e\" (UID: \"aeb32039-e94a-43e6-83d2-054fcc21849e\") " Oct 30 06:19:50.471052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.470897 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-openshift-ca\") pod \"76fef77f-35b2-4c64-a22d-293d4229df88\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " Oct 30 06:19:50.471052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.470977 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fef77f-35b2-4c64-a22d-293d4229df88-kube-api-access-4f278\" (UniqueName: \"kubernetes.io/projected/76fef77f-35b2-4c64-a22d-293d4229df88-kube-api-access-4f278\") pod \"76fef77f-35b2-4c64-a22d-293d4229df88\" (UID: \"76fef77f-35b2-4c64-a22d-293d4229df88\") " Oct 30 06:19:50.471326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.471279 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb32039-e94a-43e6-83d2-054fcc21849e-plugins-config-volume" pod "aeb32039-e94a-43e6-83d2-054fcc21849e" (UID: "aeb32039-e94a-43e6-83d2-054fcc21849e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:50.471705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.471291 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-openshift-ca" pod "76fef77f-35b2-4c64-a22d-293d4229df88" (UID: "76fef77f-35b2-4c64-a22d-293d4229df88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:50.471838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.471746 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb32039-e94a-43e6-83d2-054fcc21849e-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:50.473775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.473751 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-xks-ca" pod "76fef77f-35b2-4c64-a22d-293d4229df88" (UID: "76fef77f-35b2-4c64-a22d-293d4229df88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:50.475247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.473927 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb32039-e94a-43e6-83d2-054fcc21849e-kube-api-access-f7qbh" pod "aeb32039-e94a-43e6-83d2-054fcc21849e" (UID: "aeb32039-e94a-43e6-83d2-054fcc21849e"). InnerVolumeSpecName "kube-api-access-f7qbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:50.475247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.473963 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fef77f-35b2-4c64-a22d-293d4229df88-kube-api-access-4f278" pod "76fef77f-35b2-4c64-a22d-293d4229df88" (UID: "76fef77f-35b2-4c64-a22d-293d4229df88"). InnerVolumeSpecName "kube-api-access-4f278". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:50.572846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.572801 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:50.572846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.572836 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f278\" (UniqueName: \"kubernetes.io/projected/76fef77f-35b2-4c64-a22d-293d4229df88-kube-api-access-4f278\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:50.572846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.572849 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7qbh\" (UniqueName: \"kubernetes.io/projected/aeb32039-e94a-43e6-83d2-054fcc21849e-kube-api-access-f7qbh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:50.573089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.572862 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fef77f-35b2-4c64-a22d-293d4229df88-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:19:50.868805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.868769 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" Oct 30 06:19:50.868805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.868789 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x" event={"ID":"76fef77f-35b2-4c64-a22d-293d4229df88","Type":"ContainerDied","Data":"3b98600794df6384e1f6bf86249ddc3637fb71ac383fde7cd10f9c0fd7f8728e"} Oct 30 06:19:50.869369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.868853 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b98600794df6384e1f6bf86249ddc3637fb71ac383fde7cd10f9c0fd7f8728e" Oct 30 06:19:50.870896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.870870 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" event={"ID":"1407ec9d-8fd0-4752-90ac-26b31e3d3035","Type":"ContainerDied","Data":"c756c4f54670a197d91f73e95f31bcd67ea60237e15d5304d98582d0e7ab3934"} Oct 30 06:19:50.871043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.870918 5001 scope.go:122] "RemoveContainer" containerID="356262877e35f0271de7fed51e17ca4cee24cf876a26c723d7838cb672f5aef9" Oct 30 06:19:50.871043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.871022 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd" Oct 30 06:19:50.874255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.874231 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" Oct 30 06:19:50.874434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.874235 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c" event={"ID":"aeb32039-e94a-43e6-83d2-054fcc21849e","Type":"ContainerDied","Data":"b62cfdddf6b9de8077ffccac52ebe6a127b5be69a9997fff3741fe9eeead81a8"} Oct 30 06:19:50.885793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.885400 5001 scope.go:122] "RemoveContainer" containerID="7685a8fdb4ecc0f530f0e390bd2a722a9d7e90de1a1899a0e6a1fefd01353f9a" Oct 30 06:19:50.938800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.938764 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd"] Oct 30 06:19:50.950381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.950351 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-7fb8b5d94c-4tjvd"] Oct 30 06:19:50.993010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.992967 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c"] Oct 30 06:19:50.999973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:50.999938 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-86885bc9c4-cgd7c"] Oct 30 06:19:52.126693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:52.126659 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1407ec9d-8fd0-4752-90ac-26b31e3d3035" path="/var/lib/kubelet/pods/1407ec9d-8fd0-4752-90ac-26b31e3d3035/volumes" Oct 30 06:19:52.127102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:52.127088 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb32039-e94a-43e6-83d2-054fcc21849e" path="/var/lib/kubelet/pods/aeb32039-e94a-43e6-83d2-054fcc21849e/volumes" Oct 30 06:19:56.833482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:56.832838 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb"] Oct 30 06:19:56.834977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:56.833193 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" containerID="cri-o://6c4633cacaac77a9e4de1bb8f44c85e69307b7efc37b312db7c3fd0ff0304e7e" gracePeriod=30 Oct 30 06:19:56.841734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:56.841624 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw"] Oct 30 06:19:56.841938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:56.841909 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" containerID="cri-o://28704140959d6feed6149018ed61727fccd0d96faf5198fd0011cc1cdb2e09d8" gracePeriod=30 Oct 30 06:19:56.849699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:56.849671 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96"] Oct 30 06:19:56.849971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:56.849949 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" containerID="cri-o://c58fe15738a9549ed62b53893c4269a3f887967c028deac887d75d6db3e81830" gracePeriod=30 Oct 30 06:19:56.856390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:56.856360 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd"] Oct 30 06:19:56.856683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:56.856632 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" containerID="cri-o://3bdc572672d7c68e842a3ddbde11b7ff56200c0746529025e3797669be018705" gracePeriod=30 Oct 30 06:19:57.467587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.467543 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2"] Oct 30 06:19:57.468333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468290 5001 cpu_manager.go:414] "containerMap: removing container" podUID="aeb32039-e94a-43e6-83d2-054fcc21849e" containerName="payload-processing" Oct 30 06:19:57.468333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468322 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb32039-e94a-43e6-83d2-054fcc21849e" containerName="payload-processing" Oct 30 06:19:57.468455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468366 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1407ec9d-8fd0-4752-90ac-26b31e3d3035" containerName="payload-pre-processing" Oct 30 06:19:57.468455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468372 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1407ec9d-8fd0-4752-90ac-26b31e3d3035" containerName="payload-pre-processing" Oct 30 06:19:57.468455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468383 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b129648a-3291-4760-aa07-57f3ec743256" containerName="revoke-keys" Oct 30 06:19:57.468455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468388 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b129648a-3291-4760-aa07-57f3ec743256" containerName="revoke-keys" Oct 30 06:19:57.468455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468397 5001 cpu_manager.go:414] "containerMap: removing container" podUID="76fef77f-35b2-4c64-a22d-293d4229df88" containerName="revoke-keys" Oct 30 06:19:57.468455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468402 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76fef77f-35b2-4c64-a22d-293d4229df88" containerName="revoke-keys" Oct 30 06:19:57.468684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468501 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1407ec9d-8fd0-4752-90ac-26b31e3d3035" containerName="payload-pre-processing" Oct 30 06:19:57.468684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468512 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b129648a-3291-4760-aa07-57f3ec743256" containerName="revoke-keys" Oct 30 06:19:57.468684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468519 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="76fef77f-35b2-4c64-a22d-293d4229df88" containerName="revoke-keys" Oct 30 06:19:57.468684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.468527 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb32039-e94a-43e6-83d2-054fcc21849e" containerName="payload-processing" Oct 30 06:19:57.472186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.472157 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:19:57.486970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.486948 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:19:57.487161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.487037 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:19:57.491176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.491157 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:19:57.491476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.491457 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:19:57.494998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.494976 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2"] Oct 30 06:19:57.617234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.617200 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31407e-xk7s2\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:19:57.617490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.617269 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8nkh\" (UniqueName: \"kubernetes.io/projected/885d0da5-c3f7-49db-aa1f-b02567b3b89b-kube-api-access-z8nkh\") pod \"maas-api-revoke-keys-e2e-api-b-31407e-xk7s2\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:19:57.617490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.617476 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31407e-xk7s2\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:19:57.719351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.719219 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31407e-xk7s2\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:19:57.719537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.719383 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31407e-xk7s2\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:19:57.719537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.719440 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8nkh\" (UniqueName: \"kubernetes.io/projected/885d0da5-c3f7-49db-aa1f-b02567b3b89b-kube-api-access-z8nkh\") pod \"maas-api-revoke-keys-e2e-api-b-31407e-xk7s2\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:19:57.720125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.720097 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31407e-xk7s2\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:19:57.721986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.721966 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31407e-xk7s2\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:19:57.742820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.742792 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8nkh\" (UniqueName: \"kubernetes.io/projected/885d0da5-c3f7-49db-aa1f-b02567b3b89b-kube-api-access-z8nkh\") pod \"maas-api-revoke-keys-e2e-api-b-31407e-xk7s2\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:19:57.792227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:57.792188 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:19:58.113564 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:19:58.113528 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885d0da5_c3f7_49db_aa1f_b02567b3b89b.slice/crio-bfc56fc15e1419ea824b5dc3f20b36193e7a3ccd7ef62b71254f0125eaec43bd WatchSource:0}: Error finding container bfc56fc15e1419ea824b5dc3f20b36193e7a3ccd7ef62b71254f0125eaec43bd: Status 404 returned error can't find the container with id bfc56fc15e1419ea824b5dc3f20b36193e7a3ccd7ef62b71254f0125eaec43bd Oct 30 06:19:58.119222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:58.119198 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2"] Oct 30 06:19:58.125224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:58.125195 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:19:58.125392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:58.125248 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:19:58.125809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:58.125786 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:19:58.125866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:58.125840 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:19:58.128708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:58.128683 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:19:58.128823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:58.128729 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:19:58.986918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:58.986878 5001 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:19:58.987147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:58.986941 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:19:59.069188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:59.069146 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" event={"ID":"885d0da5-c3f7-49db-aa1f-b02567b3b89b","Type":"ContainerStarted","Data":"7322dbbe577f4e2915e5d7bdbb8fa682bfc668d838337ce369f49e5a852a3d33"} Oct 30 06:19:59.069402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:59.069204 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" event={"ID":"885d0da5-c3f7-49db-aa1f-b02567b3b89b","Type":"ContainerStarted","Data":"bfc56fc15e1419ea824b5dc3f20b36193e7a3ccd7ef62b71254f0125eaec43bd"} Oct 30 06:19:59.108215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:19:59.108160 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" podStartSLOduration=2.108145671 podStartE2EDuration="2.108145671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:59.107497092 +0000 UTC m=+1613.103454136" watchObservedRunningTime="2026-10-30 06:19:59.108145671 +0000 UTC m=+1613.104102681" Oct 30 06:20:00.082236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:00.082199 5001 generic.go:356] "Generic (PLEG): container finished" podID="885d0da5-c3f7-49db-aa1f-b02567b3b89b" containerID="7322dbbe577f4e2915e5d7bdbb8fa682bfc668d838337ce369f49e5a852a3d33" exitCode=0 Oct 30 06:20:00.082785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:00.082290 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" event={"ID":"885d0da5-c3f7-49db-aa1f-b02567b3b89b","Type":"ContainerDied","Data":"7322dbbe577f4e2915e5d7bdbb8fa682bfc668d838337ce369f49e5a852a3d33"} Oct 30 06:20:01.273352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:01.273285 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:20:01.375494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:01.375460 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-openshift-ca\") pod \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " Oct 30 06:20:01.375678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:01.375624 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885d0da5-c3f7-49db-aa1f-b02567b3b89b-kube-api-access-z8nkh\" (UniqueName: \"kubernetes.io/projected/885d0da5-c3f7-49db-aa1f-b02567b3b89b-kube-api-access-z8nkh\") pod \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " Oct 30 06:20:01.375736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:01.375711 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-xks-ca\") pod \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\" (UID: \"885d0da5-c3f7-49db-aa1f-b02567b3b89b\") " Oct 30 06:20:01.375949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:01.375917 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-openshift-ca" pod "885d0da5-c3f7-49db-aa1f-b02567b3b89b" (UID: "885d0da5-c3f7-49db-aa1f-b02567b3b89b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:01.376263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:01.376244 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.377931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:01.377900 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-xks-ca" pod "885d0da5-c3f7-49db-aa1f-b02567b3b89b" (UID: "885d0da5-c3f7-49db-aa1f-b02567b3b89b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:01.378031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:01.377955 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885d0da5-c3f7-49db-aa1f-b02567b3b89b-kube-api-access-z8nkh" pod "885d0da5-c3f7-49db-aa1f-b02567b3b89b" (UID: "885d0da5-c3f7-49db-aa1f-b02567b3b89b"). InnerVolumeSpecName "kube-api-access-z8nkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:01.478050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:01.478010 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8nkh\" (UniqueName: \"kubernetes.io/projected/885d0da5-c3f7-49db-aa1f-b02567b3b89b-kube-api-access-z8nkh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.478050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:01.478042 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885d0da5-c3f7-49db-aa1f-b02567b3b89b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.112292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.112255 5001 generic.go:356] "Generic (PLEG): container finished" podID="21141de6-8bbc-4245-9c58-d4218a32169c" containerID="28704140959d6feed6149018ed61727fccd0d96faf5198fd0011cc1cdb2e09d8" exitCode=0 Oct 30 06:20:02.114760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.114725 5001 generic.go:356] "Generic (PLEG): container finished" podID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerID="c58fe15738a9549ed62b53893c4269a3f887967c028deac887d75d6db3e81830" exitCode=0 Oct 30 06:20:02.118927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.118837 5001 generic.go:356] "Generic (PLEG): container finished" podID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerID="6c4633cacaac77a9e4de1bb8f44c85e69307b7efc37b312db7c3fd0ff0304e7e" exitCode=0 Oct 30 06:20:02.124044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.124010 5001 generic.go:356] "Generic (PLEG): container finished" podID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerID="3bdc572672d7c68e842a3ddbde11b7ff56200c0746529025e3797669be018705" exitCode=0 Oct 30 06:20:02.124382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.124354 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" event={"ID":"21141de6-8bbc-4245-9c58-d4218a32169c","Type":"ContainerDied","Data":"28704140959d6feed6149018ed61727fccd0d96faf5198fd0011cc1cdb2e09d8"} Oct 30 06:20:02.124487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.124420 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" event={"ID":"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6","Type":"ContainerDied","Data":"c58fe15738a9549ed62b53893c4269a3f887967c028deac887d75d6db3e81830"} Oct 30 06:20:02.124566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.124517 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" event={"ID":"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83","Type":"ContainerDied","Data":"6c4633cacaac77a9e4de1bb8f44c85e69307b7efc37b312db7c3fd0ff0304e7e"} Oct 30 06:20:02.124566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.124549 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" event={"ID":"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6","Type":"ContainerDied","Data":"3bdc572672d7c68e842a3ddbde11b7ff56200c0746529025e3797669be018705"} Oct 30 06:20:02.124730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.124694 5001 scope.go:122] "RemoveContainer" containerID="4523f586adf5e1f1d1d98c5cb05be0d3bda229178280f153e194ae473e2b2959" Oct 30 06:20:02.128002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.127980 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" Oct 30 06:20:02.128112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.127978 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2" event={"ID":"885d0da5-c3f7-49db-aa1f-b02567b3b89b","Type":"ContainerDied","Data":"bfc56fc15e1419ea824b5dc3f20b36193e7a3ccd7ef62b71254f0125eaec43bd"} Oct 30 06:20:02.128219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.128203 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfc56fc15e1419ea824b5dc3f20b36193e7a3ccd7ef62b71254f0125eaec43bd" Oct 30 06:20:02.162251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.162217 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:20:02.298846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.298520 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:20:02.301742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.301712 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:20:02.314694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.314650 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-credential-socket\") pod \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " Oct 30 06:20:02.314893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.314720 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-data\") pod \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " Oct 30 06:20:02.314893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.314747 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-certs\") pod \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " Oct 30 06:20:02.314893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.314779 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-socket\") pod \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " Oct 30 06:20:02.315058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.314949 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istiod-ca-cert\") pod \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " Oct 30 06:20:02.315058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.314992 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-envoy\") pod \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " Oct 30 06:20:02.315165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.315078 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-token\" (UniqueName: \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-token\") pod \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " Oct 30 06:20:02.315165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.315120 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-kube-api-access-h5kv9\" (UniqueName: \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-kube-api-access-h5kv9\") pod \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " Oct 30 06:20:02.315259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.315182 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-podinfo\") pod \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\" (UID: \"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6\") " Oct 30 06:20:02.315926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.315874 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-socket" pod "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" (UID: "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.316135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.316109 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-credential-socket" pod "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" (UID: "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.316365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.316343 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-data" pod "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" (UID: "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.316546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.316525 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-certs" pod "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" (UID: "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.317347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.317274 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istiod-ca-cert" pod "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" (UID: "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:02.319609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.319582 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-token" pod "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" (UID: "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.319729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.319629 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-podinfo" pod "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" (UID: "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:02.319962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.319946 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-kube-api-access-h5kv9" pod "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" (UID: "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6"). InnerVolumeSpecName "kube-api-access-h5kv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.321599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.321574 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-envoy" pod "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" (UID: "45d0d063-92c4-4dd1-a1d4-a546bbd47eb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.362380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.362278 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:20:02.416534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416502 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-socket\") pod \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " Oct 30 06:20:02.416721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416597 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-data\") pod \"21141de6-8bbc-4245-9c58-d4218a32169c\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " Oct 30 06:20:02.416721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416667 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-podinfo\") pod \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " Oct 30 06:20:02.416721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416704 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21141de6-8bbc-4245-9c58-d4218a32169c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21141de6-8bbc-4245-9c58-d4218a32169c-istio-podinfo\") pod \"21141de6-8bbc-4245-9c58-d4218a32169c\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " Oct 30 06:20:02.416926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416740 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-envoy\") pod \"21141de6-8bbc-4245-9c58-d4218a32169c\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " Oct 30 06:20:02.416926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416774 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-certs\") pod \"21141de6-8bbc-4245-9c58-d4218a32169c\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " Oct 30 06:20:02.416926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416837 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istiod-ca-cert\") pod \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " Oct 30 06:20:02.416926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416871 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-certs\") pod \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " Oct 30 06:20:02.416926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416905 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-kube-api-access-68r4b\" (UniqueName: \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-kube-api-access-68r4b\") pod \"21141de6-8bbc-4245-9c58-d4218a32169c\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " Oct 30 06:20:02.417171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416913 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-data" pod "21141de6-8bbc-4245-9c58-d4218a32169c" (UID: "21141de6-8bbc-4245-9c58-d4218a32169c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.417171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416922 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-socket" pod "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" (UID: "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.417171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.416943 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-credential-socket\") pod \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " Oct 30 06:20:02.417171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.417075 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21141de6-8bbc-4245-9c58-d4218a32169c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21141de6-8bbc-4245-9c58-d4218a32169c-istiod-ca-cert\") pod \"21141de6-8bbc-4245-9c58-d4218a32169c\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " Oct 30 06:20:02.417171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.417117 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-socket\") pod \"21141de6-8bbc-4245-9c58-d4218a32169c\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " Oct 30 06:20:02.417171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.417164 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-data\") pod \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " Oct 30 06:20:02.417493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.417161 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-certs" pod "21141de6-8bbc-4245-9c58-d4218a32169c" (UID: "21141de6-8bbc-4245-9c58-d4218a32169c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.417493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.417203 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-credential-socket\") pod \"21141de6-8bbc-4245-9c58-d4218a32169c\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " Oct 30 06:20:02.417493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.417237 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-certs" pod "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" (UID: "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.417493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.417252 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-istio-token\" (UniqueName: \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-istio-token\") pod \"21141de6-8bbc-4245-9c58-d4218a32169c\" (UID: \"21141de6-8bbc-4245-9c58-d4218a32169c\") " Oct 30 06:20:02.417493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.417415 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-envoy\") pod \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " Oct 30 06:20:02.417493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.417455 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-kube-api-access-s7jxt\" (UniqueName: \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-kube-api-access-s7jxt\") pod \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " Oct 30 06:20:02.417493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.417491 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-token\" (UniqueName: \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-token\") pod \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\" (UID: \"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83\") " Oct 30 06:20:02.417827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.417625 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istiod-ca-cert" pod "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" (UID: "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:02.418300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.418274 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-data" pod "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" (UID: "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.418565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.418542 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-credential-socket" pod "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" (UID: "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.418909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.418884 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.418917 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.418932 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.418945 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.418956 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.418968 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.418982 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.418993 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.419005 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.419016 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.419030 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5kv9\" (UniqueName: \"kubernetes.io/projected/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-kube-api-access-h5kv9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.419109 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.419123 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.419136 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.419441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.419224 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21141de6-8bbc-4245-9c58-d4218a32169c-istiod-ca-cert" pod "21141de6-8bbc-4245-9c58-d4218a32169c" (UID: "21141de6-8bbc-4245-9c58-d4218a32169c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:02.419744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.419601 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-credential-socket" pod "21141de6-8bbc-4245-9c58-d4218a32169c" (UID: "21141de6-8bbc-4245-9c58-d4218a32169c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.419802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.419758 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-socket" pod "21141de6-8bbc-4245-9c58-d4218a32169c" (UID: "21141de6-8bbc-4245-9c58-d4218a32169c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.421616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.421575 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-envoy" pod "21141de6-8bbc-4245-9c58-d4218a32169c" (UID: "21141de6-8bbc-4245-9c58-d4218a32169c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.422547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.422525 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-kube-api-access-68r4b" pod "21141de6-8bbc-4245-9c58-d4218a32169c" (UID: "21141de6-8bbc-4245-9c58-d4218a32169c"). InnerVolumeSpecName "kube-api-access-68r4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.426799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.426691 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-podinfo" pod "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" (UID: "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:02.426799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.426764 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-envoy" pod "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" (UID: "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.426925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.426804 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-kube-api-access-s7jxt" pod "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" (UID: "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83"). InnerVolumeSpecName "kube-api-access-s7jxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.426925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.426830 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-token" pod "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" (UID: "d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.427258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.427233 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21141de6-8bbc-4245-9c58-d4218a32169c-istio-podinfo" pod "21141de6-8bbc-4245-9c58-d4218a32169c" (UID: "21141de6-8bbc-4245-9c58-d4218a32169c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:02.428601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.428577 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-istio-token" pod "21141de6-8bbc-4245-9c58-d4218a32169c" (UID: "21141de6-8bbc-4245-9c58-d4218a32169c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.520550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.520509 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-envoy\") pod \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " Oct 30 06:20:02.520723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.520603 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-kube-api-access-mg48b\" (UniqueName: \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-kube-api-access-mg48b\") pod \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " Oct 30 06:20:02.520723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.520690 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-certs\") pod \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " Oct 30 06:20:02.520864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.520847 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-socket\") pod \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " Oct 30 06:20:02.520956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.520895 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istiod-ca-cert\") pod \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " Oct 30 06:20:02.520956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.520928 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-data\") pod \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " Oct 30 06:20:02.521060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.520970 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-token\" (UniqueName: \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-token\") pod \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " Oct 30 06:20:02.521060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.520987 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-certs" pod "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" (UID: "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.521060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521029 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-credential-socket\") pod \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " Oct 30 06:20:02.521283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521094 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-podinfo\") pod \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\" (UID: \"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6\") " Oct 30 06:20:02.521283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521088 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-socket" pod "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" (UID: "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.521283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521217 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-data" pod "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" (UID: "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.521283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521248 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istiod-ca-cert" pod "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" (UID: "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521469 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-credential-socket" pod "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" (UID: "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521745 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68r4b\" (UniqueName: \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-kube-api-access-68r4b\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521766 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521780 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21141de6-8bbc-4245-9c58-d4218a32169c-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521793 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521805 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521817 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521830 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521854 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21141de6-8bbc-4245-9c58-d4218a32169c-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521866 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521878 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521890 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jxt\" (UniqueName: \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-kube-api-access-s7jxt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521903 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521915 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521927 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521938 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521951 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521963 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21141de6-8bbc-4245-9c58-d4218a32169c-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.522094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.521975 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21141de6-8bbc-4245-9c58-d4218a32169c-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.523375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.523354 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-token" pod "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" (UID: "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.523547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.523527 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-envoy" pod "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" (UID: "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.523605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.523554 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-kube-api-access-mg48b" pod "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" (UID: "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6"). InnerVolumeSpecName "kube-api-access-mg48b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.523838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.523822 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-podinfo" pod "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" (UID: "382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:02.623083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.622977 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.623083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.623016 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg48b\" (UniqueName: \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-kube-api-access-mg48b\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.623083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.623032 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.623083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:02.623045 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:03.139534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.139505 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" Oct 30 06:20:03.139718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.139506 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd" event={"ID":"382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6","Type":"ContainerDied","Data":"45500014d991344b1bea96e5524b8784032ebeaf3809a665bae41537e7db2cb8"} Oct 30 06:20:03.139718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.139672 5001 scope.go:122] "RemoveContainer" containerID="3bdc572672d7c68e842a3ddbde11b7ff56200c0746529025e3797669be018705" Oct 30 06:20:03.141948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.141922 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" Oct 30 06:20:03.142066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.141924 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw" event={"ID":"21141de6-8bbc-4245-9c58-d4218a32169c","Type":"ContainerDied","Data":"5d1b36a83565190e2224e6aad01ff75b547b6b2bbda7760b309cec064360de3f"} Oct 30 06:20:03.143951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.143927 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" Oct 30 06:20:03.143951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.143939 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96" event={"ID":"45d0d063-92c4-4dd1-a1d4-a546bbd47eb6","Type":"ContainerDied","Data":"6d5b4323feb55ba97bad6a0dbdea6ac443746afc029e7161e74c95b7697ac060"} Oct 30 06:20:03.145930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.145907 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" event={"ID":"d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83","Type":"ContainerDied","Data":"55171cdd5fc62d5ce89eaf039fdf9963f3f9ad7920f010c48c2066751eb7f92e"} Oct 30 06:20:03.146048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.146020 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb" Oct 30 06:20:03.155980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.155837 5001 scope.go:122] "RemoveContainer" containerID="28704140959d6feed6149018ed61727fccd0d96faf5198fd0011cc1cdb2e09d8" Oct 30 06:20:03.168296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.168270 5001 scope.go:122] "RemoveContainer" containerID="c58fe15738a9549ed62b53893c4269a3f887967c028deac887d75d6db3e81830" Oct 30 06:20:03.180038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.180011 5001 scope.go:122] "RemoveContainer" containerID="6c4633cacaac77a9e4de1bb8f44c85e69307b7efc37b312db7c3fd0ff0304e7e" Oct 30 06:20:03.237218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.237188 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb"] Oct 30 06:20:03.248099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.248067 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-6zhnb"] Oct 30 06:20:03.273544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.273508 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd"] Oct 30 06:20:03.277977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.277946 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-t9kjd"] Oct 30 06:20:03.311322 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.311273 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw"] Oct 30 06:20:03.317135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.317105 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-wfhsw"] Oct 30 06:20:03.352463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.352428 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96"] Oct 30 06:20:03.359225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:03.359188 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-jrn96"] Oct 30 06:20:04.120575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:04.120541 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" path="/var/lib/kubelet/pods/21141de6-8bbc-4245-9c58-d4218a32169c/volumes" Oct 30 06:20:04.121114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:04.121094 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" path="/var/lib/kubelet/pods/382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6/volumes" Oct 30 06:20:04.121582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:04.121567 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" path="/var/lib/kubelet/pods/45d0d063-92c4-4dd1-a1d4-a546bbd47eb6/volumes" Oct 30 06:20:04.122296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:04.122282 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" path="/var/lib/kubelet/pods/d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83/volumes" Oct 30 06:20:05.411808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:05.411772 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x"] Oct 30 06:20:05.417538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:05.417510 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-584c9d-kq28x"] Oct 30 06:20:06.121857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:06.121818 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76fef77f-35b2-4c64-a22d-293d4229df88" path="/var/lib/kubelet/pods/76fef77f-35b2-4c64-a22d-293d4229df88/volumes" Oct 30 06:20:07.845494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:07.845466 5001 scope.go:122] "RemoveContainer" containerID="83b0b94a7f62bf7a2f544a30b1f46e03c84d3043ea6d1e56c8d72aed3edbb720" Oct 30 06:20:07.864253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:07.864225 5001 scope.go:122] "RemoveContainer" containerID="8c8a53770c1339dd9208d82dcbd200fed9f045ab3680e2a8b4d2a64b24a2f94e" Oct 30 06:20:07.876977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:07.876955 5001 scope.go:122] "RemoveContainer" containerID="9c2a7f92cd843105b66f457a01beec73dd17a8409fb7ecc469bb74a91632c10a" Oct 30 06:20:07.887643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:07.887614 5001 scope.go:122] "RemoveContainer" containerID="f03ddb2377c8dde2df34e0cd7d376224aa3ad3bb5cf1809288ce356b2f4d84d2" Oct 30 06:20:07.917724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:07.917702 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz"] Oct 30 06:20:08.294437 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:08.294276 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:20:12.900193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:12.900159 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf"] Oct 30 06:20:12.900872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:12.900432 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" containerID="cri-o://0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f" gracePeriod=30 Oct 30 06:20:12.918720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:12.918686 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" podUID="ce3d7158-7803-403a-b160-da80e0d7a684" containerName="payload-processing" containerID="cri-o://86a42b51b3021d2b08b8460cd17e2a59cc002a4cdadb44476a48e4f831c7ea3f" gracePeriod=25 Oct 30 06:20:13.697562 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.697526 5001 generic.go:356] "Generic (PLEG): container finished" podID="ce3d7158-7803-403a-b160-da80e0d7a684" containerID="86a42b51b3021d2b08b8460cd17e2a59cc002a4cdadb44476a48e4f831c7ea3f" exitCode=0 Oct 30 06:20:13.697784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.697668 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" event={"ID":"ce3d7158-7803-403a-b160-da80e0d7a684","Type":"ContainerDied","Data":"86a42b51b3021d2b08b8460cd17e2a59cc002a4cdadb44476a48e4f831c7ea3f"} Oct 30 06:20:13.700144 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.700118 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt"] Oct 30 06:20:13.701257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701230 5001 cpu_manager.go:414] "containerMap: removing container" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" Oct 30 06:20:13.701257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701253 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" Oct 30 06:20:13.701506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701295 5001 cpu_manager.go:414] "containerMap: removing container" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" Oct 30 06:20:13.701506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701318 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" Oct 30 06:20:13.701506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701339 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" Oct 30 06:20:13.701506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701346 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" Oct 30 06:20:13.701506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701360 5001 cpu_manager.go:414] "containerMap: removing container" podUID="885d0da5-c3f7-49db-aa1f-b02567b3b89b" containerName="revoke-keys" Oct 30 06:20:13.701506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701366 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885d0da5-c3f7-49db-aa1f-b02567b3b89b" containerName="revoke-keys" Oct 30 06:20:13.701506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701389 5001 cpu_manager.go:414] "containerMap: removing container" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" Oct 30 06:20:13.701506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701396 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" Oct 30 06:20:13.701506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701417 5001 cpu_manager.go:414] "containerMap: removing container" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" Oct 30 06:20:13.701506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701425 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" Oct 30 06:20:13.702058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701567 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a09c60-c8bc-4c25-b3ca-b4aa0123fd83" containerName="istio-proxy" Oct 30 06:20:13.702058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701582 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="885d0da5-c3f7-49db-aa1f-b02567b3b89b" containerName="revoke-keys" Oct 30 06:20:13.702058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701593 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d0d063-92c4-4dd1-a1d4-a546bbd47eb6" containerName="istio-proxy" Oct 30 06:20:13.702058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701605 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" Oct 30 06:20:13.702058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.701615 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="382c841d-ed3a-43fe-bd5f-76a1ba0aaeb6" containerName="istio-proxy" Oct 30 06:20:13.704176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.704155 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:20:13.705679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.705659 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:20:13.705822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.705752 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt"] Oct 30 06:20:13.712163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.712135 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc"] Oct 30 06:20:13.712994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.712973 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ce3d7158-7803-403a-b160-da80e0d7a684" containerName="payload-processing" Oct 30 06:20:13.713121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.712995 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3d7158-7803-403a-b160-da80e0d7a684" containerName="payload-processing" Oct 30 06:20:13.713219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.713205 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="21141de6-8bbc-4245-9c58-d4218a32169c" containerName="istio-proxy" Oct 30 06:20:13.713278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.713227 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3d7158-7803-403a-b160-da80e0d7a684" containerName="payload-processing" Oct 30 06:20:13.716845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.716827 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:13.717974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.717952 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc"] Oct 30 06:20:13.720634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.720612 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bc5d8f\"" Oct 30 06:20:13.720734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.720707 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bc5d8f-dockercfg-4xd59\"" Oct 30 06:20:13.721826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.721806 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm"] Oct 30 06:20:13.727016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.726995 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm"] Oct 30 06:20:13.727111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.727098 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:20:13.731511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.731492 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:20:13.731601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.731571 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:20:13.731601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.731572 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:20:13.731690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.731597 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:20:13.762559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.762514 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm9vt\" (UniqueName: \"kubernetes.io/projected/98e8a39c-1675-4737-9e04-c740cec60898-kube-api-access-qm9vt\") pod \"maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:13.762760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.762673 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsp86\" (UniqueName: \"kubernetes.io/projected/64a0b9ed-f74f-4647-9956-664b0924db28-kube-api-access-lsp86\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt\" (UID: \"64a0b9ed-f74f-4647-9956-664b0924db28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:20:13.762760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.762739 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ks6p\" (UniqueName: \"kubernetes.io/projected/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-kube-api-access-9ks6p\") pod \"payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm\" (UID: \"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:20:13.762889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.762792 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:13.762889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.762839 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:13.763037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.763016 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a0b9ed-f74f-4647-9956-664b0924db28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt\" (UID: \"64a0b9ed-f74f-4647-9956-664b0924db28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:20:13.763076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.763052 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm\" (UID: \"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:20:13.863614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.863582 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3d7158-7803-403a-b160-da80e0d7a684-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3d7158-7803-403a-b160-da80e0d7a684-plugins-config-volume\") pod \"ce3d7158-7803-403a-b160-da80e0d7a684\" (UID: \"ce3d7158-7803-403a-b160-da80e0d7a684\") " Oct 30 06:20:13.863801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.863742 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3d7158-7803-403a-b160-da80e0d7a684-kube-api-access-5r27z\" (UniqueName: \"kubernetes.io/projected/ce3d7158-7803-403a-b160-da80e0d7a684-kube-api-access-5r27z\") pod \"ce3d7158-7803-403a-b160-da80e0d7a684\" (UID: \"ce3d7158-7803-403a-b160-da80e0d7a684\") " Oct 30 06:20:13.863923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.863888 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3d7158-7803-403a-b160-da80e0d7a684-plugins-config-volume" pod "ce3d7158-7803-403a-b160-da80e0d7a684" (UID: "ce3d7158-7803-403a-b160-da80e0d7a684"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:13.864019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.863968 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a0b9ed-f74f-4647-9956-664b0924db28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt\" (UID: \"64a0b9ed-f74f-4647-9956-664b0924db28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:20:13.864019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.863989 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm\" (UID: \"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:20:13.864117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.864056 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm9vt\" (UniqueName: \"kubernetes.io/projected/98e8a39c-1675-4737-9e04-c740cec60898-kube-api-access-qm9vt\") pod \"maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:13.864168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.864144 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsp86\" (UniqueName: \"kubernetes.io/projected/64a0b9ed-f74f-4647-9956-664b0924db28-kube-api-access-lsp86\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt\" (UID: \"64a0b9ed-f74f-4647-9956-664b0924db28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:20:13.864229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.864183 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ks6p\" (UniqueName: \"kubernetes.io/projected/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-kube-api-access-9ks6p\") pod \"payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm\" (UID: \"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:20:13.864281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.864242 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:13.864377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.864291 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:13.864477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.864460 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3d7158-7803-403a-b160-da80e0d7a684-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:13.864894 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.864864 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a0b9ed-f74f-4647-9956-664b0924db28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt\" (UID: \"64a0b9ed-f74f-4647-9956-664b0924db28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:20:13.865035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.864953 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm\" (UID: \"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:20:13.865035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.864985 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:13.866248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.866225 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3d7158-7803-403a-b160-da80e0d7a684-kube-api-access-5r27z" pod "ce3d7158-7803-403a-b160-da80e0d7a684" (UID: "ce3d7158-7803-403a-b160-da80e0d7a684"). InnerVolumeSpecName "kube-api-access-5r27z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:13.866945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.866929 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:13.889074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.889053 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsp86\" (UniqueName: \"kubernetes.io/projected/64a0b9ed-f74f-4647-9956-664b0924db28-kube-api-access-lsp86\") pod \"payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt\" (UID: \"64a0b9ed-f74f-4647-9956-664b0924db28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:20:13.889182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.889055 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm9vt\" (UniqueName: \"kubernetes.io/projected/98e8a39c-1675-4737-9e04-c740cec60898-kube-api-access-qm9vt\") pod \"maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:13.889182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.889090 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ks6p\" (UniqueName: \"kubernetes.io/projected/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-kube-api-access-9ks6p\") pod \"payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm\" (UID: \"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:20:13.965611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:13.965528 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r27z\" (UniqueName: \"kubernetes.io/projected/ce3d7158-7803-403a-b160-da80e0d7a684-kube-api-access-5r27z\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:14.022984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.022944 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:20:14.032541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.032515 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:14.045919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.045889 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:20:14.381001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.380944 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt"] Oct 30 06:20:14.384400 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:20:14.384371 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a0b9ed_f74f_4647_9956_664b0924db28.slice/crio-56e790caeef92b2657f1580961fa378fd04ecde0094f8241f2f6d69685e42007 WatchSource:0}: Error finding container 56e790caeef92b2657f1580961fa378fd04ecde0094f8241f2f6d69685e42007: Status 404 returned error can't find the container with id 56e790caeef92b2657f1580961fa378fd04ecde0094f8241f2f6d69685e42007 Oct 30 06:20:14.403738 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:20:14.403707 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e8a39c_1675_4737_9e04_c740cec60898.slice/crio-8418da3e80a02cb0331b7cf27aca7dc75ada56e642a77415c9e094e324210c64 WatchSource:0}: Error finding container 8418da3e80a02cb0331b7cf27aca7dc75ada56e642a77415c9e094e324210c64: Status 404 returned error can't find the container with id 8418da3e80a02cb0331b7cf27aca7dc75ada56e642a77415c9e094e324210c64 Oct 30 06:20:14.407120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.407094 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc"] Oct 30 06:20:14.419143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.419122 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm"] Oct 30 06:20:14.420193 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:20:14.420168 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d46490_0091_4cb4_a80e_cd13b7ec2ea9.slice/crio-4aa5a198ed655e21d4471d559dde75b341996a2dae80b6225679c4f1b8fb2c42 WatchSource:0}: Error finding container 4aa5a198ed655e21d4471d559dde75b341996a2dae80b6225679c4f1b8fb2c42: Status 404 returned error can't find the container with id 4aa5a198ed655e21d4471d559dde75b341996a2dae80b6225679c4f1b8fb2c42 Oct 30 06:20:14.708812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.708715 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" event={"ID":"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9","Type":"ContainerStarted","Data":"023e9bced6c183cb8a3dddbddf14e9ff6608181b358edd38c5a7f58e5cd4d875"} Oct 30 06:20:14.708812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.708776 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" event={"ID":"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9","Type":"ContainerStarted","Data":"4aa5a198ed655e21d4471d559dde75b341996a2dae80b6225679c4f1b8fb2c42"} Oct 30 06:20:14.708812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.708812 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:20:14.710420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.710399 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" Oct 30 06:20:14.710566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.710415 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz" event={"ID":"ce3d7158-7803-403a-b160-da80e0d7a684","Type":"ContainerDied","Data":"381651531cf0dd6d377ca9b8e1e3233b4efbb022b4a4ec63f46d7bf3303f17c8"} Oct 30 06:20:14.710566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.710476 5001 scope.go:122] "RemoveContainer" containerID="86a42b51b3021d2b08b8460cd17e2a59cc002a4cdadb44476a48e4f831c7ea3f" Oct 30 06:20:14.713597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.713529 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" event={"ID":"98e8a39c-1675-4737-9e04-c740cec60898","Type":"ContainerStarted","Data":"8418da3e80a02cb0331b7cf27aca7dc75ada56e642a77415c9e094e324210c64"} Oct 30 06:20:14.715478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.715456 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" event={"ID":"64a0b9ed-f74f-4647-9956-664b0924db28","Type":"ContainerStarted","Data":"223b961376126b550517270b5241b7b9b447d855ab80b3c9eeb760610dbda399"} Oct 30 06:20:14.715591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.715493 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" event={"ID":"64a0b9ed-f74f-4647-9956-664b0924db28","Type":"ContainerStarted","Data":"56e790caeef92b2657f1580961fa378fd04ecde0094f8241f2f6d69685e42007"} Oct 30 06:20:14.715662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.715596 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:20:14.800187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.800127 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" podStartSLOduration=1.800107318 podStartE2EDuration="1.800107318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:14.795560333 +0000 UTC m=+1628.791517343" watchObservedRunningTime="2026-10-30 06:20:14.800107318 +0000 UTC m=+1628.796064331" Oct 30 06:20:14.840954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.840899 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" podStartSLOduration=1.840883641 podStartE2EDuration="1.840883641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:14.83806374 +0000 UTC m=+1628.834020752" watchObservedRunningTime="2026-10-30 06:20:14.840883641 +0000 UTC m=+1628.836840650" Oct 30 06:20:14.876795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.876756 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz"] Oct 30 06:20:14.883572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:14.883539 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31407e-78c4f67fb8-nkhjz"] Oct 30 06:20:15.727090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:15.726995 5001 generic.go:356] "Generic (PLEG): container finished" podID="98e8a39c-1675-4737-9e04-c740cec60898" containerID="c0436674e950e530290a77401cf4594858d0e23328b107f8f3b27402e245dedc" exitCode=0 Oct 30 06:20:15.727090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:15.727073 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" event={"ID":"98e8a39c-1675-4737-9e04-c740cec60898","Type":"ContainerDied","Data":"c0436674e950e530290a77401cf4594858d0e23328b107f8f3b27402e245dedc"} Oct 30 06:20:16.120433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:16.120400 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3d7158-7803-403a-b160-da80e0d7a684" path="/var/lib/kubelet/pods/ce3d7158-7803-403a-b160-da80e0d7a684/volumes" Oct 30 06:20:16.915106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:16.915076 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:17.041002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.040913 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e8a39c-1675-4737-9e04-c740cec60898-kube-api-access-qm9vt\" (UniqueName: \"kubernetes.io/projected/98e8a39c-1675-4737-9e04-c740cec60898-kube-api-access-qm9vt\") pod \"98e8a39c-1675-4737-9e04-c740cec60898\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " Oct 30 06:20:17.041002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.040957 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-xks-ca\") pod \"98e8a39c-1675-4737-9e04-c740cec60898\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " Oct 30 06:20:17.041250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.041221 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-openshift-ca\") pod \"98e8a39c-1675-4737-9e04-c740cec60898\" (UID: \"98e8a39c-1675-4737-9e04-c740cec60898\") " Oct 30 06:20:17.041619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.041582 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-openshift-ca" pod "98e8a39c-1675-4737-9e04-c740cec60898" (UID: "98e8a39c-1675-4737-9e04-c740cec60898"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:17.041890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.041793 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:17.043282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.043258 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-xks-ca" pod "98e8a39c-1675-4737-9e04-c740cec60898" (UID: "98e8a39c-1675-4737-9e04-c740cec60898"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:17.043397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.043385 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e8a39c-1675-4737-9e04-c740cec60898-kube-api-access-qm9vt" pod "98e8a39c-1675-4737-9e04-c740cec60898" (UID: "98e8a39c-1675-4737-9e04-c740cec60898"). InnerVolumeSpecName "kube-api-access-qm9vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:17.143202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.143167 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm9vt\" (UniqueName: \"kubernetes.io/projected/98e8a39c-1675-4737-9e04-c740cec60898-kube-api-access-qm9vt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:17.143202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.143200 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e8a39c-1675-4737-9e04-c740cec60898-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:17.748325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.748282 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" Oct 30 06:20:17.748535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.748289 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc" event={"ID":"98e8a39c-1675-4737-9e04-c740cec60898","Type":"ContainerDied","Data":"8418da3e80a02cb0331b7cf27aca7dc75ada56e642a77415c9e094e324210c64"} Oct 30 06:20:17.748535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:17.748371 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8418da3e80a02cb0331b7cf27aca7dc75ada56e642a77415c9e094e324210c64" Oct 30 06:20:18.196987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.196958 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:20:18.258450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.256963 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-credential-socket\") pod \"5da87567-9dee-47ec-9bfd-52f7cb332923\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " Oct 30 06:20:18.258450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.257045 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-data\") pod \"5da87567-9dee-47ec-9bfd-52f7cb332923\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " Oct 30 06:20:18.258450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.257091 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da87567-9dee-47ec-9bfd-52f7cb332923-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da87567-9dee-47ec-9bfd-52f7cb332923-istiod-ca-cert\") pod \"5da87567-9dee-47ec-9bfd-52f7cb332923\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " Oct 30 06:20:18.258450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.257163 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-envoy\") pod \"5da87567-9dee-47ec-9bfd-52f7cb332923\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " Oct 30 06:20:18.258450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.257237 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-kube-api-access-g52lc\" (UniqueName: \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-kube-api-access-g52lc\") pod \"5da87567-9dee-47ec-9bfd-52f7cb332923\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " Oct 30 06:20:18.258450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.257373 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-socket\") pod \"5da87567-9dee-47ec-9bfd-52f7cb332923\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " Oct 30 06:20:18.258450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.257472 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-token\" (UniqueName: \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-token\") pod \"5da87567-9dee-47ec-9bfd-52f7cb332923\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " Oct 30 06:20:18.258450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.257515 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-podinfo\") pod \"5da87567-9dee-47ec-9bfd-52f7cb332923\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " Oct 30 06:20:18.258450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.257550 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-certs\") pod \"5da87567-9dee-47ec-9bfd-52f7cb332923\" (UID: \"5da87567-9dee-47ec-9bfd-52f7cb332923\") " Oct 30 06:20:18.259041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.258841 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-credential-socket" pod "5da87567-9dee-47ec-9bfd-52f7cb332923" (UID: "5da87567-9dee-47ec-9bfd-52f7cb332923"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:18.259620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.259367 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-data" pod "5da87567-9dee-47ec-9bfd-52f7cb332923" (UID: "5da87567-9dee-47ec-9bfd-52f7cb332923"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:18.261351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.261327 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da87567-9dee-47ec-9bfd-52f7cb332923-istiod-ca-cert" pod "5da87567-9dee-47ec-9bfd-52f7cb332923" (UID: "5da87567-9dee-47ec-9bfd-52f7cb332923"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:18.264230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.264202 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-token" pod "5da87567-9dee-47ec-9bfd-52f7cb332923" (UID: "5da87567-9dee-47ec-9bfd-52f7cb332923"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:18.264458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.264440 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-socket" pod "5da87567-9dee-47ec-9bfd-52f7cb332923" (UID: "5da87567-9dee-47ec-9bfd-52f7cb332923"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:18.267629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.267602 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-certs" pod "5da87567-9dee-47ec-9bfd-52f7cb332923" (UID: "5da87567-9dee-47ec-9bfd-52f7cb332923"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:18.267957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.267937 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-envoy" pod "5da87567-9dee-47ec-9bfd-52f7cb332923" (UID: "5da87567-9dee-47ec-9bfd-52f7cb332923"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:18.270268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.270201 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-podinfo" pod "5da87567-9dee-47ec-9bfd-52f7cb332923" (UID: "5da87567-9dee-47ec-9bfd-52f7cb332923"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:18.271702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.271615 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-kube-api-access-g52lc" pod "5da87567-9dee-47ec-9bfd-52f7cb332923" (UID: "5da87567-9dee-47ec-9bfd-52f7cb332923"). InnerVolumeSpecName "kube-api-access-g52lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:18.360139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.360096 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:18.360139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.360131 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:18.360139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.360140 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:18.360139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.360150 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:18.360502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.360158 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:18.360502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.360166 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:18.360502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.360175 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da87567-9dee-47ec-9bfd-52f7cb332923-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:18.360502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.360182 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da87567-9dee-47ec-9bfd-52f7cb332923-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:18.360502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.360190 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g52lc\" (UniqueName: \"kubernetes.io/projected/5da87567-9dee-47ec-9bfd-52f7cb332923-kube-api-access-g52lc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:18.759070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.759035 5001 generic.go:356] "Generic (PLEG): container finished" podID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerID="0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f" exitCode=0 Oct 30 06:20:18.759264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.759106 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" Oct 30 06:20:18.759264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.759113 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" event={"ID":"5da87567-9dee-47ec-9bfd-52f7cb332923","Type":"ContainerDied","Data":"0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f"} Oct 30 06:20:18.759264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.759167 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf" event={"ID":"5da87567-9dee-47ec-9bfd-52f7cb332923","Type":"ContainerDied","Data":"a666ecac18f820d8b593271b43608fa58521c0d87ca1d57bab6a8a1d820689d3"} Oct 30 06:20:18.759264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.759204 5001 scope.go:122] "RemoveContainer" containerID="0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f" Oct 30 06:20:18.774063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.774041 5001 scope.go:122] "RemoveContainer" containerID="2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d" Oct 30 06:20:18.785705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.785680 5001 scope.go:122] "RemoveContainer" containerID="0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f" Oct 30 06:20:18.786084 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:18.786045 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f\": container with ID starting with 0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f not found: ID does not exist" containerID="0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f" Oct 30 06:20:18.786180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.786087 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f"} err="failed to get container status \"0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f\": rpc error: code = NotFound desc = could not find container \"0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f\": container with ID starting with 0aa9fcde02d8bc8a8df67330de0351675cecde37bd90bb74c14e86ed9b2b391f not found: ID does not exist" Oct 30 06:20:18.786180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.786113 5001 scope.go:122] "RemoveContainer" containerID="2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d" Oct 30 06:20:18.786447 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:18.786420 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d\": container with ID starting with 2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d not found: ID does not exist" containerID="2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d" Oct 30 06:20:18.786517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.786457 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d"} err="failed to get container status \"2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d\": rpc error: code = NotFound desc = could not find container \"2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d\": container with ID starting with 2c2c8ed485b4e000ec6f7ad4095a519863da970a81dcf6199a80dfc7b4a9b01d not found: ID does not exist" Oct 30 06:20:18.824397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.824363 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf"] Oct 30 06:20:18.828233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:18.828209 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-584c9d-openshift-default-7756f8d445-96rdf"] Oct 30 06:20:20.117742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:20.117702 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" path="/var/lib/kubelet/pods/5da87567-9dee-47ec-9bfd-52f7cb332923/volumes" Oct 30 06:20:22.758260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:22.758223 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j"] Oct 30 06:20:22.764126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:22.764088 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c99c-6gl8j"] Oct 30 06:20:23.920378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:23.912993 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:20:24.119845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:24.119793 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b129648a-3291-4760-aa07-57f3ec743256" path="/var/lib/kubelet/pods/b129648a-3291-4760-aa07-57f3ec743256/volumes" Oct 30 06:20:25.729979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:25.729950 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:20:25.730494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:25.730021 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:20:27.115284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:27.115249 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl"] Oct 30 06:20:27.115736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:27.115435 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" podUID="2dfaafe5-2935-4184-8c71-78644c4f7d5d" containerName="payload-processing" containerID="cri-o://4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b" gracePeriod=30 Oct 30 06:20:27.149601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:27.148666 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97"] Oct 30 06:20:27.149601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:27.148935 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" podUID="7bc81a5c-ef66-4f6d-80f5-65abb5264d82" containerName="payload-pre-processing" containerID="cri-o://23fcfa351b696218279e552fd565d6592c137c35b7c6b35a172fc88370772ad6" gracePeriod=30 Oct 30 06:20:27.818163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:27.818097 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc"] Oct 30 06:20:27.823268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:27.822932 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:20:27.830004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:27.829893 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8272af-b7zvc"] Oct 30 06:20:28.028495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:27.965639 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dfaafe5-2935-4184-8c71-78644c4f7d5d-kube-api-access-7d996\" (UniqueName: \"kubernetes.io/projected/2dfaafe5-2935-4184-8c71-78644c4f7d5d-kube-api-access-7d996\") pod \"2dfaafe5-2935-4184-8c71-78644c4f7d5d\" (UID: \"2dfaafe5-2935-4184-8c71-78644c4f7d5d\") " Oct 30 06:20:28.028495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:27.965879 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dfaafe5-2935-4184-8c71-78644c4f7d5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfaafe5-2935-4184-8c71-78644c4f7d5d-plugins-config-volume\") pod \"2dfaafe5-2935-4184-8c71-78644c4f7d5d\" (UID: \"2dfaafe5-2935-4184-8c71-78644c4f7d5d\") " Oct 30 06:20:28.029593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.029536 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dfaafe5-2935-4184-8c71-78644c4f7d5d-plugins-config-volume" pod "2dfaafe5-2935-4184-8c71-78644c4f7d5d" (UID: "2dfaafe5-2935-4184-8c71-78644c4f7d5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:28.042651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.040351 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94"] Oct 30 06:20:28.042651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.041349 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" Oct 30 06:20:28.042651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.041370 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" Oct 30 06:20:28.042651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.041386 5001 cpu_manager.go:414] "containerMap: removing container" podUID="98e8a39c-1675-4737-9e04-c740cec60898" containerName="revoke-keys" Oct 30 06:20:28.042651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.041393 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e8a39c-1675-4737-9e04-c740cec60898" containerName="revoke-keys" Oct 30 06:20:28.042651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.041460 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2dfaafe5-2935-4184-8c71-78644c4f7d5d" containerName="payload-processing" Oct 30 06:20:28.042651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.041469 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dfaafe5-2935-4184-8c71-78644c4f7d5d" containerName="payload-processing" Oct 30 06:20:28.042651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.041637 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" Oct 30 06:20:28.042651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.041647 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dfaafe5-2935-4184-8c71-78644c4f7d5d" containerName="payload-processing" Oct 30 06:20:28.042651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.041665 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" Oct 30 06:20:28.042651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.041673 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e8a39c-1675-4737-9e04-c740cec60898" containerName="revoke-keys" Oct 30 06:20:28.044852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.044818 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfaafe5-2935-4184-8c71-78644c4f7d5d-kube-api-access-7d996" pod "2dfaafe5-2935-4184-8c71-78644c4f7d5d" (UID: "2dfaafe5-2935-4184-8c71-78644c4f7d5d"). InnerVolumeSpecName "kube-api-access-7d996". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:28.048352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.047827 5001 generic.go:356] "Generic (PLEG): container finished" podID="7bc81a5c-ef66-4f6d-80f5-65abb5264d82" containerID="23fcfa351b696218279e552fd565d6592c137c35b7c6b35a172fc88370772ad6" exitCode=0 Oct 30 06:20:28.048943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.048702 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94"] Oct 30 06:20:28.048943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.048745 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" event={"ID":"7bc81a5c-ef66-4f6d-80f5-65abb5264d82","Type":"ContainerDied","Data":"23fcfa351b696218279e552fd565d6592c137c35b7c6b35a172fc88370772ad6"} Oct 30 06:20:28.048943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.048847 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:28.075374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.050695 5001 generic.go:356] "Generic (PLEG): container finished" podID="2dfaafe5-2935-4184-8c71-78644c4f7d5d" containerID="4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b" exitCode=0 Oct 30 06:20:28.075374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.050740 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" event={"ID":"2dfaafe5-2935-4184-8c71-78644c4f7d5d","Type":"ContainerDied","Data":"4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b"} Oct 30 06:20:28.075374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.050772 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" event={"ID":"2dfaafe5-2935-4184-8c71-78644c4f7d5d","Type":"ContainerDied","Data":"acac8653399751e602eaee95a103502a7fdcb5feca0db65288220661586459d7"} Oct 30 06:20:28.075374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.050792 5001 scope.go:122] "RemoveContainer" containerID="4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b" Oct 30 06:20:28.075374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.050896 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl" Oct 30 06:20:28.110391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.087002 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:20:28.110391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.087072 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:20:28.110391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.087205 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:20:28.110391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.087914 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:20:28.129754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.129374 5001 scope.go:122] "RemoveContainer" containerID="4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b" Oct 30 06:20:28.129754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.129561 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:28.129754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.129613 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:28.130183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.129852 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2clv\" (UniqueName: \"kubernetes.io/projected/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-kube-api-access-m2clv\") pod \"maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:28.130183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.130083 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d996\" (UniqueName: \"kubernetes.io/projected/2dfaafe5-2935-4184-8c71-78644c4f7d5d-kube-api-access-7d996\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.130183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.130107 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfaafe5-2935-4184-8c71-78644c4f7d5d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.130183 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:28.130146 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b\": container with ID starting with 4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b not found: ID does not exist" containerID="4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b" Oct 30 06:20:28.130730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.130176 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b"} err="failed to get container status \"4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b\": rpc error: code = NotFound desc = could not find container \"4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b\": container with ID starting with 4c24aa62bbb0812b9f8cd24d921c417eb26840a78236b14a82cf2cbd40c33a1b not found: ID does not exist" Oct 30 06:20:28.131532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.131511 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e8a39c-1675-4737-9e04-c740cec60898" path="/var/lib/kubelet/pods/98e8a39c-1675-4737-9e04-c740cec60898/volumes" Oct 30 06:20:28.199507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.199470 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl"] Oct 30 06:20:28.200822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.200737 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:20:28.205251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.204337 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-7979ff4776-bqhjl"] Oct 30 06:20:28.231242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.231204 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:28.231534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.231513 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:28.231873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.231851 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:28.231988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.231864 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2clv\" (UniqueName: \"kubernetes.io/projected/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-kube-api-access-m2clv\") pod \"maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:28.236340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.234607 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:28.297842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.297742 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2clv\" (UniqueName: \"kubernetes.io/projected/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-kube-api-access-m2clv\") pod \"maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:28.332550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.332522 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-plugins-config-volume\") pod \"7bc81a5c-ef66-4f6d-80f5-65abb5264d82\" (UID: \"7bc81a5c-ef66-4f6d-80f5-65abb5264d82\") " Oct 30 06:20:28.332764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.332653 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-kube-api-access-7hglk\" (UniqueName: \"kubernetes.io/projected/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-kube-api-access-7hglk\") pod \"7bc81a5c-ef66-4f6d-80f5-65abb5264d82\" (UID: \"7bc81a5c-ef66-4f6d-80f5-65abb5264d82\") " Oct 30 06:20:28.333228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.333200 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-plugins-config-volume" pod "7bc81a5c-ef66-4f6d-80f5-65abb5264d82" (UID: "7bc81a5c-ef66-4f6d-80f5-65abb5264d82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:28.335189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.335163 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-kube-api-access-7hglk" pod "7bc81a5c-ef66-4f6d-80f5-65abb5264d82" (UID: "7bc81a5c-ef66-4f6d-80f5-65abb5264d82"). InnerVolumeSpecName "kube-api-access-7hglk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:28.434246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.434208 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hglk\" (UniqueName: \"kubernetes.io/projected/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-kube-api-access-7hglk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.434246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.434243 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc81a5c-ef66-4f6d-80f5-65abb5264d82-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.494147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.493930 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:28.876408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:28.876281 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94"] Oct 30 06:20:28.895524 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:20:28.894846 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c9a551_5a64_4ff7_a465_5b0e29ba004c.slice/crio-788deb18f4e155f583c9c3cc032537fa718828cca350b4887bb5010dfe605ed0 WatchSource:0}: Error finding container 788deb18f4e155f583c9c3cc032537fa718828cca350b4887bb5010dfe605ed0: Status 404 returned error can't find the container with id 788deb18f4e155f583c9c3cc032537fa718828cca350b4887bb5010dfe605ed0 Oct 30 06:20:29.096202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:29.096158 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" event={"ID":"7bc81a5c-ef66-4f6d-80f5-65abb5264d82","Type":"ContainerDied","Data":"6a2bf1d11ae6f8e81df1a8af3698a6ee36726e69edd548e2e94d532471f26787"} Oct 30 06:20:29.096444 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:29.096240 5001 scope.go:122] "RemoveContainer" containerID="23fcfa351b696218279e552fd565d6592c137c35b7c6b35a172fc88370772ad6" Oct 30 06:20:29.098801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:29.098767 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" event={"ID":"b6c9a551-5a64-4ff7-a465-5b0e29ba004c","Type":"ContainerStarted","Data":"788deb18f4e155f583c9c3cc032537fa718828cca350b4887bb5010dfe605ed0"} Oct 30 06:20:29.101123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:29.101099 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97" Oct 30 06:20:29.168635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:29.157255 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97"] Oct 30 06:20:29.168635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:29.161939 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-b7cbd96bc-r4f97"] Oct 30 06:20:30.128774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:30.128736 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dfaafe5-2935-4184-8c71-78644c4f7d5d" path="/var/lib/kubelet/pods/2dfaafe5-2935-4184-8c71-78644c4f7d5d/volumes" Oct 30 06:20:30.129483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:30.129462 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc81a5c-ef66-4f6d-80f5-65abb5264d82" path="/var/lib/kubelet/pods/7bc81a5c-ef66-4f6d-80f5-65abb5264d82/volumes" Oct 30 06:20:30.134389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:30.134361 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" event={"ID":"b6c9a551-5a64-4ff7-a465-5b0e29ba004c","Type":"ContainerStarted","Data":"32e849a8114adad7db011e86447967e144c6bd5428ce3f53cf4ea7b9f392a688"} Oct 30 06:20:30.188333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:30.185850 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" podStartSLOduration=3.18582812 podStartE2EDuration="3.18582812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:30.179741218 +0000 UTC m=+1644.175698228" watchObservedRunningTime="2026-10-30 06:20:30.18582812 +0000 UTC m=+1644.181785132" Oct 30 06:20:30.231647 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:30.231606 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c9a551_5a64_4ff7_a465_5b0e29ba004c.slice/crio-conmon-32e849a8114adad7db011e86447967e144c6bd5428ce3f53cf4ea7b9f392a688.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c9a551_5a64_4ff7_a465_5b0e29ba004c.slice/crio-32e849a8114adad7db011e86447967e144c6bd5428ce3f53cf4ea7b9f392a688.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:20:31.120012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:31.119978 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl"] Oct 30 06:20:31.120238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:31.120207 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" containerID="cri-o://a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09" gracePeriod=30 Oct 30 06:20:31.145049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:31.145019 5001 generic.go:356] "Generic (PLEG): container finished" podID="b6c9a551-5a64-4ff7-a465-5b0e29ba004c" containerID="32e849a8114adad7db011e86447967e144c6bd5428ce3f53cf4ea7b9f392a688" exitCode=0 Oct 30 06:20:31.145221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:31.145104 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" event={"ID":"b6c9a551-5a64-4ff7-a465-5b0e29ba004c","Type":"ContainerDied","Data":"32e849a8114adad7db011e86447967e144c6bd5428ce3f53cf4ea7b9f392a688"} Oct 30 06:20:32.347523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:32.347495 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:32.382213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:32.382176 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-xks-ca\") pod \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " Oct 30 06:20:32.382399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:32.382231 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-openshift-ca\") pod \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " Oct 30 06:20:32.382619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:32.382596 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-kube-api-access-m2clv\" (UniqueName: \"kubernetes.io/projected/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-kube-api-access-m2clv\") pod \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\" (UID: \"b6c9a551-5a64-4ff7-a465-5b0e29ba004c\") " Oct 30 06:20:32.382752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:32.382715 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-openshift-ca" pod "b6c9a551-5a64-4ff7-a465-5b0e29ba004c" (UID: "b6c9a551-5a64-4ff7-a465-5b0e29ba004c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:32.383133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:32.383113 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:32.384938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:32.384906 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-kube-api-access-m2clv" pod "b6c9a551-5a64-4ff7-a465-5b0e29ba004c" (UID: "b6c9a551-5a64-4ff7-a465-5b0e29ba004c"). InnerVolumeSpecName "kube-api-access-m2clv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:32.385059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:32.385030 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-xks-ca" pod "b6c9a551-5a64-4ff7-a465-5b0e29ba004c" (UID: "b6c9a551-5a64-4ff7-a465-5b0e29ba004c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:32.484679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:32.484582 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2clv\" (UniqueName: \"kubernetes.io/projected/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-kube-api-access-m2clv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:32.484679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:32.484616 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c9a551-5a64-4ff7-a465-5b0e29ba004c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:33.168058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:33.168010 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" event={"ID":"b6c9a551-5a64-4ff7-a465-5b0e29ba004c","Type":"ContainerDied","Data":"788deb18f4e155f583c9c3cc032537fa718828cca350b4887bb5010dfe605ed0"} Oct 30 06:20:33.168341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:33.168082 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="788deb18f4e155f583c9c3cc032537fa718828cca350b4887bb5010dfe605ed0" Oct 30 06:20:33.168341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:33.168156 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94" Oct 30 06:20:33.370497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:33.370457 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:20:33.370937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:33.370523 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:33.781866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:33.781829 5001 patch_prober.go:29] interesting pod/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:20:33.782045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:33.781892 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:20:34.137643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:34.137600 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w"] Oct 30 06:20:36.135008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.134977 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j"] Oct 30 06:20:36.135855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.135823 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" containerID="cri-o://da48dd99c1ee6e995240909fd08725313186c9afdd385858b5df069b79f2eb6a" gracePeriod=30 Oct 30 06:20:36.300220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.300183 5001 generic.go:356] "Generic (PLEG): container finished" podID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerID="7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e" exitCode=0 Oct 30 06:20:36.300773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.300736 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" event={"ID":"10785405-d520-4f7f-9ddc-af67c7c5a1bf","Type":"ContainerDied","Data":"7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e"} Oct 30 06:20:36.301636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.301608 5001 scope.go:122] "RemoveContainer" containerID="7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e" Oct 30 06:20:36.317928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.317875 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:36.332525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.332478 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:36.332823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.332798 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:36.627786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.627752 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:20:36.758922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.758814 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-data\") pod \"20c3319d-5388-4262-8abc-3bd921947b04\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " Oct 30 06:20:36.759122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.759101 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-istio-token\" (UniqueName: \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-istio-token\") pod \"20c3319d-5388-4262-8abc-3bd921947b04\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " Oct 30 06:20:36.759206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.759109 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-data" pod "20c3319d-5388-4262-8abc-3bd921947b04" (UID: "20c3319d-5388-4262-8abc-3bd921947b04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:36.759206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.759146 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-kube-api-access-pj4cg\" (UniqueName: \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-kube-api-access-pj4cg\") pod \"20c3319d-5388-4262-8abc-3bd921947b04\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " Oct 30 06:20:36.759206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.759195 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20c3319d-5388-4262-8abc-3bd921947b04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c3319d-5388-4262-8abc-3bd921947b04-istio-podinfo\") pod \"20c3319d-5388-4262-8abc-3bd921947b04\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " Oct 30 06:20:36.759406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.759231 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-envoy\") pod \"20c3319d-5388-4262-8abc-3bd921947b04\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " Oct 30 06:20:36.759406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.759373 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-certs\") pod \"20c3319d-5388-4262-8abc-3bd921947b04\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " Oct 30 06:20:36.759516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.759408 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c3319d-5388-4262-8abc-3bd921947b04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c3319d-5388-4262-8abc-3bd921947b04-istiod-ca-cert\") pod \"20c3319d-5388-4262-8abc-3bd921947b04\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " Oct 30 06:20:36.759516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.759441 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-credential-socket\") pod \"20c3319d-5388-4262-8abc-3bd921947b04\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " Oct 30 06:20:36.759516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.759473 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-socket\") pod \"20c3319d-5388-4262-8abc-3bd921947b04\" (UID: \"20c3319d-5388-4262-8abc-3bd921947b04\") " Oct 30 06:20:36.760117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.759743 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-certs" pod "20c3319d-5388-4262-8abc-3bd921947b04" (UID: "20c3319d-5388-4262-8abc-3bd921947b04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:36.760117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.759980 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-socket" pod "20c3319d-5388-4262-8abc-3bd921947b04" (UID: "20c3319d-5388-4262-8abc-3bd921947b04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:36.760117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.760065 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-credential-socket" pod "20c3319d-5388-4262-8abc-3bd921947b04" (UID: "20c3319d-5388-4262-8abc-3bd921947b04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:36.760117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.760078 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c3319d-5388-4262-8abc-3bd921947b04-istiod-ca-cert" pod "20c3319d-5388-4262-8abc-3bd921947b04" (UID: "20c3319d-5388-4262-8abc-3bd921947b04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:36.760386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.760128 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:36.760386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.760148 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:36.760386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.760161 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:36.762247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.762222 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-envoy" pod "20c3319d-5388-4262-8abc-3bd921947b04" (UID: "20c3319d-5388-4262-8abc-3bd921947b04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:36.762584 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.762563 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-istio-token" pod "20c3319d-5388-4262-8abc-3bd921947b04" (UID: "20c3319d-5388-4262-8abc-3bd921947b04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:36.762676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.762614 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-kube-api-access-pj4cg" pod "20c3319d-5388-4262-8abc-3bd921947b04" (UID: "20c3319d-5388-4262-8abc-3bd921947b04"). InnerVolumeSpecName "kube-api-access-pj4cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:36.762933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.762914 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20c3319d-5388-4262-8abc-3bd921947b04-istio-podinfo" pod "20c3319d-5388-4262-8abc-3bd921947b04" (UID: "20c3319d-5388-4262-8abc-3bd921947b04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:36.862245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.862205 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c3319d-5388-4262-8abc-3bd921947b04-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:36.862245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.862249 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:36.862487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.862267 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:36.862487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.862282 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4cg\" (UniqueName: \"kubernetes.io/projected/20c3319d-5388-4262-8abc-3bd921947b04-kube-api-access-pj4cg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:36.862487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.862295 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c3319d-5388-4262-8abc-3bd921947b04-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:36.862487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:36.862333 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c3319d-5388-4262-8abc-3bd921947b04-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:37.018863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.018780 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8"] Oct 30 06:20:37.019472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019457 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" Oct 30 06:20:37.019522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019473 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da87567-9dee-47ec-9bfd-52f7cb332923" containerName="istio-proxy" Oct 30 06:20:37.019522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019507 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b6c9a551-5a64-4ff7-a465-5b0e29ba004c" containerName="revoke-keys" Oct 30 06:20:37.019522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019512 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c9a551-5a64-4ff7-a465-5b0e29ba004c" containerName="revoke-keys" Oct 30 06:20:37.019616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019529 5001 cpu_manager.go:414] "containerMap: removing container" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" Oct 30 06:20:37.019616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019534 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" Oct 30 06:20:37.019616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019542 5001 cpu_manager.go:414] "containerMap: removing container" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" Oct 30 06:20:37.019616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019547 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" Oct 30 06:20:37.019616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019555 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7bc81a5c-ef66-4f6d-80f5-65abb5264d82" containerName="payload-pre-processing" Oct 30 06:20:37.019616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019560 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc81a5c-ef66-4f6d-80f5-65abb5264d82" containerName="payload-pre-processing" Oct 30 06:20:37.019792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019653 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c9a551-5a64-4ff7-a465-5b0e29ba004c" containerName="revoke-keys" Oct 30 06:20:37.019792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019663 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" Oct 30 06:20:37.019792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019668 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c3319d-5388-4262-8abc-3bd921947b04" containerName="istio-proxy" Oct 30 06:20:37.019792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.019674 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc81a5c-ef66-4f6d-80f5-65abb5264d82" containerName="payload-pre-processing" Oct 30 06:20:37.022349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.022331 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:37.038996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.038967 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:20:37.039171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.039030 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:20:37.043553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.043529 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8"] Oct 30 06:20:37.079330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.074022 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vwx\" (UniqueName: \"kubernetes.io/projected/93e99c8e-12fc-4412-9cb9-4612ad6869d9-kube-api-access-x4vwx\") pod \"maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:37.079330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.074196 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:37.079330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.074390 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:37.079330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.076095 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:20:37.079330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.076233 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:20:37.177344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.177173 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:37.177344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.177243 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vwx\" (UniqueName: \"kubernetes.io/projected/93e99c8e-12fc-4412-9cb9-4612ad6869d9-kube-api-access-x4vwx\") pod \"maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:37.177949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.177401 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:37.179336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.179134 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:37.182413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.182388 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:37.203658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.203629 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vwx\" (UniqueName: \"kubernetes.io/projected/93e99c8e-12fc-4412-9cb9-4612ad6869d9-kube-api-access-x4vwx\") pod \"maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:37.311278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.311186 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" event={"ID":"10785405-d520-4f7f-9ddc-af67c7c5a1bf","Type":"ContainerStarted","Data":"03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d"} Oct 30 06:20:37.313447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.313421 5001 generic.go:356] "Generic (PLEG): container finished" podID="20c3319d-5388-4262-8abc-3bd921947b04" containerID="a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09" exitCode=0 Oct 30 06:20:37.313616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.313467 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" Oct 30 06:20:37.313616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.313503 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" event={"ID":"20c3319d-5388-4262-8abc-3bd921947b04","Type":"ContainerDied","Data":"a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09"} Oct 30 06:20:37.313616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.313570 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl" event={"ID":"20c3319d-5388-4262-8abc-3bd921947b04","Type":"ContainerDied","Data":"eef16cd1a8025342e7f01a92b959a19137e0a395c6ce04dbe7c709cdcd1e9f27"} Oct 30 06:20:37.313616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.313603 5001 scope.go:122] "RemoveContainer" containerID="a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09" Oct 30 06:20:37.325406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.325378 5001 scope.go:122] "RemoveContainer" containerID="a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71" Oct 30 06:20:37.335900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.335868 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:37.336483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.336464 5001 scope.go:122] "RemoveContainer" containerID="a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09" Oct 30 06:20:37.336832 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:37.336811 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09\": container with ID starting with a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09 not found: ID does not exist" containerID="a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09" Oct 30 06:20:37.336899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.336843 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09"} err="failed to get container status \"a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09\": rpc error: code = NotFound desc = could not find container \"a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09\": container with ID starting with a4e0a0ed828850bb149e510b0fbda9c80e8d6e21e6d6744df0650cd50c30ad09 not found: ID does not exist" Oct 30 06:20:37.336899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.336862 5001 scope.go:122] "RemoveContainer" containerID="a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71" Oct 30 06:20:37.337104 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:37.337088 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71\": container with ID starting with a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71 not found: ID does not exist" containerID="a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71" Oct 30 06:20:37.337192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.337108 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71"} err="failed to get container status \"a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71\": rpc error: code = NotFound desc = could not find container \"a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71\": container with ID starting with a4f24d5dd96b44aabdabb21128296b52aef89840c386d18d395edd3b24f3ce71 not found: ID does not exist" Oct 30 06:20:37.350051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.350015 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:20:37.350290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.350267 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:20:37.350393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.350369 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 06:20:37.415980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.415943 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl"] Oct 30 06:20:37.419934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.419896 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-33c99c-openshift-default-66f697958c-spgdl"] Oct 30 06:20:37.522428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.522132 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:20:37.522428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.522180 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:20:37.690772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:37.690731 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8"] Oct 30 06:20:37.692524 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:20:37.692496 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e99c8e_12fc_4412_9cb9_4612ad6869d9.slice/crio-1e263f9b697bf0f0060a48f0ff9007ca76d8eed33f3090a0d068e894cd7a031e WatchSource:0}: Error finding container 1e263f9b697bf0f0060a48f0ff9007ca76d8eed33f3090a0d068e894cd7a031e: Status 404 returned error can't find the container with id 1e263f9b697bf0f0060a48f0ff9007ca76d8eed33f3090a0d068e894cd7a031e Oct 30 06:20:38.118885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.118848 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c3319d-5388-4262-8abc-3bd921947b04" path="/var/lib/kubelet/pods/20c3319d-5388-4262-8abc-3bd921947b04/volumes" Oct 30 06:20:38.298198 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:38.298157 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:20:38.332564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.329138 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2"] Oct 30 06:20:38.334193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.333810 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" event={"ID":"93e99c8e-12fc-4412-9cb9-4612ad6869d9","Type":"ContainerStarted","Data":"baa0e95a0663b26bc4747fecd65af0ea8e2b3a7ff68a1e53c29b94743e8261e0"} Oct 30 06:20:38.334193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.333850 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" event={"ID":"93e99c8e-12fc-4412-9cb9-4612ad6869d9","Type":"ContainerStarted","Data":"1e263f9b697bf0f0060a48f0ff9007ca76d8eed33f3090a0d068e894cd7a031e"} Oct 30 06:20:38.334193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.333891 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.349381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.348625 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-520bc4-openshift-default-dockercfg-jm7pw\"" Oct 30 06:20:38.350286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.349914 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:20:38.350286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.349954 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 06:20:38.400593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.398392 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssd8j\" (UniqueName: \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-kube-api-access-ssd8j\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.400593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.398560 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-podinfo\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.400593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.398640 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-certs\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.400593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.398667 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-token\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.400593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.398696 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-credential-socket\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.402333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.402268 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975f7621-69e1-4a2c-8f93-871c39b4ca75-istiod-ca-cert\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.403793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.402627 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-envoy\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.403793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.402671 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-socket\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.403793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.402851 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-data\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.450219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.450170 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2"] Oct 30 06:20:38.497366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.495659 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" podStartSLOduration=2.495639379 podStartE2EDuration="2.495639379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:38.380477351 +0000 UTC m=+1652.376434367" watchObservedRunningTime="2026-10-30 06:20:38.495639379 +0000 UTC m=+1652.491596418" Oct 30 06:20:38.526332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.519809 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-certs\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.526332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.519865 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-token\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.526332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.519894 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-credential-socket\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.526332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.519960 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975f7621-69e1-4a2c-8f93-871c39b4ca75-istiod-ca-cert\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.526332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.520026 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-envoy\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.526332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.520058 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-socket\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.526332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.520281 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-data\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.526332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.520377 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssd8j\" (UniqueName: \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-kube-api-access-ssd8j\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.526332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.520537 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-podinfo\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.592362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.550615 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-podinfo\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.592362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.551117 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-certs\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.592362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.551713 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-credential-socket\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.592362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.552374 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975f7621-69e1-4a2c-8f93-871c39b4ca75-istiod-ca-cert\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.618856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.598148 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-socket\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.618856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.598432 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-data\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.685408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.682648 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-envoy\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.687764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.687724 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-token\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:38.752331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:38.749038 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssd8j\" (UniqueName: \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-kube-api-access-ssd8j\") pod \"e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:39.041954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.041914 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:39.138490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.138452 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" podUID="88a23f0c-7108-43e4-aa56-06005da64513" containerName="payload-processing" containerID="cri-o://a0218856d4e45c043f86615c391a4b91763c02a4c568527f8c20c99b44eac5dd" gracePeriod=25 Oct 30 06:20:39.344706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.344667 5001 generic.go:356] "Generic (PLEG): container finished" podID="93e99c8e-12fc-4412-9cb9-4612ad6869d9" containerID="baa0e95a0663b26bc4747fecd65af0ea8e2b3a7ff68a1e53c29b94743e8261e0" exitCode=0 Oct 30 06:20:39.345214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.344803 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" event={"ID":"93e99c8e-12fc-4412-9cb9-4612ad6869d9","Type":"ContainerDied","Data":"baa0e95a0663b26bc4747fecd65af0ea8e2b3a7ff68a1e53c29b94743e8261e0"} Oct 30 06:20:39.347843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.347815 5001 generic.go:356] "Generic (PLEG): container finished" podID="88a23f0c-7108-43e4-aa56-06005da64513" containerID="a0218856d4e45c043f86615c391a4b91763c02a4c568527f8c20c99b44eac5dd" exitCode=0 Oct 30 06:20:39.348139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.348120 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" event={"ID":"88a23f0c-7108-43e4-aa56-06005da64513","Type":"ContainerDied","Data":"a0218856d4e45c043f86615c391a4b91763c02a4c568527f8c20c99b44eac5dd"} Oct 30 06:20:39.350709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.350681 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:20:39.350842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.350735 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 06:20:39.429004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.422006 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2"] Oct 30 06:20:39.440221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.440176 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:39.440461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.440435 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:39.440632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.440612 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:39.515021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.514995 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:20:39.604916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.604884 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a23f0c-7108-43e4-aa56-06005da64513-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a23f0c-7108-43e4-aa56-06005da64513-plugins-config-volume\") pod \"88a23f0c-7108-43e4-aa56-06005da64513\" (UID: \"88a23f0c-7108-43e4-aa56-06005da64513\") " Oct 30 06:20:39.605094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.605073 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a23f0c-7108-43e4-aa56-06005da64513-kube-api-access-sxjg5\" (UniqueName: \"kubernetes.io/projected/88a23f0c-7108-43e4-aa56-06005da64513-kube-api-access-sxjg5\") pod \"88a23f0c-7108-43e4-aa56-06005da64513\" (UID: \"88a23f0c-7108-43e4-aa56-06005da64513\") " Oct 30 06:20:39.606608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.606576 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a23f0c-7108-43e4-aa56-06005da64513-plugins-config-volume" pod "88a23f0c-7108-43e4-aa56-06005da64513" (UID: "88a23f0c-7108-43e4-aa56-06005da64513"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:39.610351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.610280 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a23f0c-7108-43e4-aa56-06005da64513-kube-api-access-sxjg5" pod "88a23f0c-7108-43e4-aa56-06005da64513" (UID: "88a23f0c-7108-43e4-aa56-06005da64513"). InnerVolumeSpecName "kube-api-access-sxjg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:39.709360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.707640 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a23f0c-7108-43e4-aa56-06005da64513-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:39.709360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:39.707672 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxjg5\" (UniqueName: \"kubernetes.io/projected/88a23f0c-7108-43e4-aa56-06005da64513-kube-api-access-sxjg5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:40.356390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.349837 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:20:40.356390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.349917 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 06:20:40.401116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.401083 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" event={"ID":"975f7621-69e1-4a2c-8f93-871c39b4ca75","Type":"ContainerStarted","Data":"bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b"} Oct 30 06:20:40.401401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.401386 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" event={"ID":"975f7621-69e1-4a2c-8f93-871c39b4ca75","Type":"ContainerStarted","Data":"5a306016b6fab41406897b94e968f7b5908d88d647b54535e86cd8a951a432b3"} Oct 30 06:20:40.407148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.404827 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" Oct 30 06:20:40.407148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.406161 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w" event={"ID":"88a23f0c-7108-43e4-aa56-06005da64513","Type":"ContainerDied","Data":"1dd8b4282fc3866db756efdaae64c5e22972170ef83a121a184630534e68fc09"} Oct 30 06:20:40.407148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.406231 5001 scope.go:122] "RemoveContainer" containerID="a0218856d4e45c043f86615c391a4b91763c02a4c568527f8c20c99b44eac5dd" Oct 30 06:20:40.479329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.462403 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" podStartSLOduration=2.462379459 podStartE2EDuration="2.462379459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:40.451706953 +0000 UTC m=+1654.447663974" watchObservedRunningTime="2026-10-30 06:20:40.462379459 +0000 UTC m=+1654.458336469" Oct 30 06:20:40.498195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.498158 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w"] Oct 30 06:20:40.508525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.508495 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-84c74d6b8b-lcq4w"] Oct 30 06:20:40.527388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.526395 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s"] Oct 30 06:20:40.527388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.526642 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" podUID="f21ab9cb-4b51-467e-83d9-dbb0f9386660" containerName="payload-pre-processing" containerID="cri-o://4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f" gracePeriod=30 Oct 30 06:20:40.697928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.697901 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:40.831839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.827262 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e99c8e-12fc-4412-9cb9-4612ad6869d9-kube-api-access-x4vwx\" (UniqueName: \"kubernetes.io/projected/93e99c8e-12fc-4412-9cb9-4612ad6869d9-kube-api-access-x4vwx\") pod \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " Oct 30 06:20:40.831839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.827826 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-openshift-ca\") pod \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " Oct 30 06:20:40.831839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.828169 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-openshift-ca" pod "93e99c8e-12fc-4412-9cb9-4612ad6869d9" (UID: "93e99c8e-12fc-4412-9cb9-4612ad6869d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:40.831839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.828243 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-xks-ca\") pod \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\" (UID: \"93e99c8e-12fc-4412-9cb9-4612ad6869d9\") " Oct 30 06:20:40.831839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.829203 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:40.831839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.831155 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e99c8e-12fc-4412-9cb9-4612ad6869d9-kube-api-access-x4vwx" pod "93e99c8e-12fc-4412-9cb9-4612ad6869d9" (UID: "93e99c8e-12fc-4412-9cb9-4612ad6869d9"). InnerVolumeSpecName "kube-api-access-x4vwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:40.833170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.833139 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-xks-ca" pod "93e99c8e-12fc-4412-9cb9-4612ad6869d9" (UID: "93e99c8e-12fc-4412-9cb9-4612ad6869d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:40.951388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.943494 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e99c8e-12fc-4412-9cb9-4612ad6869d9-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:40.951388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:40.943532 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4vwx\" (UniqueName: \"kubernetes.io/projected/93e99c8e-12fc-4412-9cb9-4612ad6869d9-kube-api-access-x4vwx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.205275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.083490 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:41.245450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.245419 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:41.395429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.375321 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:20:41.397202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.396258 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 06:20:41.400516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.379830 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94"] Oct 30 06:20:41.400879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.400840 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f50f6e-9qj94"] Oct 30 06:20:41.405260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.405230 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:20:41.416487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.416438 5001 generic.go:356] "Generic (PLEG): container finished" podID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerID="da48dd99c1ee6e995240909fd08725313186c9afdd385858b5df069b79f2eb6a" exitCode=0 Oct 30 06:20:41.416606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.416516 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" event={"ID":"02cdba52-fb35-42fa-91f2-77ecec35b5ef","Type":"ContainerDied","Data":"da48dd99c1ee6e995240909fd08725313186c9afdd385858b5df069b79f2eb6a"} Oct 30 06:20:41.416606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.416589 5001 scope.go:122] "RemoveContainer" containerID="e5f44e7b4363c1a0ad75843d9feb6b8c070f25f0212576bfa3681c2b90f5100f" Oct 30 06:20:41.418047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.417981 5001 generic.go:356] "Generic (PLEG): container finished" podID="f21ab9cb-4b51-467e-83d9-dbb0f9386660" containerID="4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f" exitCode=0 Oct 30 06:20:41.418047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.418023 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" Oct 30 06:20:41.418143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.418062 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" event={"ID":"f21ab9cb-4b51-467e-83d9-dbb0f9386660","Type":"ContainerDied","Data":"4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f"} Oct 30 06:20:41.418143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.418114 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s" event={"ID":"f21ab9cb-4b51-467e-83d9-dbb0f9386660","Type":"ContainerDied","Data":"51c2163792ab89a07969a4ed6ebbc481868b4ec412d0f57f5946d05bc5709832"} Oct 30 06:20:41.419624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.419604 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" event={"ID":"93e99c8e-12fc-4412-9cb9-4612ad6869d9","Type":"ContainerDied","Data":"1e263f9b697bf0f0060a48f0ff9007ca76d8eed33f3090a0d068e894cd7a031e"} Oct 30 06:20:41.419712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.419638 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e263f9b697bf0f0060a48f0ff9007ca76d8eed33f3090a0d068e894cd7a031e" Oct 30 06:20:41.419712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.419696 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8" Oct 30 06:20:41.422896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.422875 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:41.424564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.424539 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:20:41.436605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.436582 5001 scope.go:122] "RemoveContainer" containerID="4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f" Oct 30 06:20:41.449468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.449439 5001 scope.go:122] "RemoveContainer" containerID="4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f" Oct 30 06:20:41.449836 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:41.449813 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f\": container with ID starting with 4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f not found: ID does not exist" containerID="4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f" Oct 30 06:20:41.449930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.449851 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f"} err="failed to get container status \"4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f\": rpc error: code = NotFound desc = could not find container \"4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f\": container with ID starting with 4943c9a36ea98e8eadd8cded1c634022ce19795b35a02c6aa1c5e2791b650b0f not found: ID does not exist" Oct 30 06:20:41.602132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.601770 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21ab9cb-4b51-467e-83d9-dbb0f9386660-kube-api-access-wv4mr\" (UniqueName: \"kubernetes.io/projected/f21ab9cb-4b51-467e-83d9-dbb0f9386660-kube-api-access-wv4mr\") pod \"f21ab9cb-4b51-467e-83d9-dbb0f9386660\" (UID: \"f21ab9cb-4b51-467e-83d9-dbb0f9386660\") " Oct 30 06:20:41.602132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.601954 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21ab9cb-4b51-467e-83d9-dbb0f9386660-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21ab9cb-4b51-467e-83d9-dbb0f9386660-plugins-config-volume\") pod \"f21ab9cb-4b51-467e-83d9-dbb0f9386660\" (UID: \"f21ab9cb-4b51-467e-83d9-dbb0f9386660\") " Oct 30 06:20:41.606675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.606636 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21ab9cb-4b51-467e-83d9-dbb0f9386660-plugins-config-volume" pod "f21ab9cb-4b51-467e-83d9-dbb0f9386660" (UID: "f21ab9cb-4b51-467e-83d9-dbb0f9386660"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:41.612981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.612942 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21ab9cb-4b51-467e-83d9-dbb0f9386660-kube-api-access-wv4mr" pod "f21ab9cb-4b51-467e-83d9-dbb0f9386660" (UID: "f21ab9cb-4b51-467e-83d9-dbb0f9386660"). InnerVolumeSpecName "kube-api-access-wv4mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:41.707846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.707816 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21ab9cb-4b51-467e-83d9-dbb0f9386660-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.707846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.707844 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv4mr\" (UniqueName: \"kubernetes.io/projected/f21ab9cb-4b51-467e-83d9-dbb0f9386660-kube-api-access-wv4mr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.727959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.727930 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:20:41.809402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.809268 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-token\" (UniqueName: \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-token\") pod \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " Oct 30 06:20:41.809829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.809802 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-credential-socket\") pod \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " Oct 30 06:20:41.809957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.809900 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-kube-api-access-bp5gj\" (UniqueName: \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-kube-api-access-bp5gj\") pod \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " Oct 30 06:20:41.809957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.809943 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-envoy\") pod \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " Oct 30 06:20:41.810351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.810111 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-credential-socket" pod "02cdba52-fb35-42fa-91f2-77ecec35b5ef" (UID: "02cdba52-fb35-42fa-91f2-77ecec35b5ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:41.810351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.810240 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-socket\") pod \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " Oct 30 06:20:41.810508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.810402 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-data\") pod \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " Oct 30 06:20:41.810508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.810436 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istiod-ca-cert\") pod \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " Oct 30 06:20:41.810625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.810567 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-podinfo\") pod \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " Oct 30 06:20:41.810682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.810619 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-socket" pod "02cdba52-fb35-42fa-91f2-77ecec35b5ef" (UID: "02cdba52-fb35-42fa-91f2-77ecec35b5ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:41.810682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.810640 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-certs\") pod \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\" (UID: \"02cdba52-fb35-42fa-91f2-77ecec35b5ef\") " Oct 30 06:20:41.811011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.810985 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istiod-ca-cert" pod "02cdba52-fb35-42fa-91f2-77ecec35b5ef" (UID: "02cdba52-fb35-42fa-91f2-77ecec35b5ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:41.811914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.811171 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-certs" pod "02cdba52-fb35-42fa-91f2-77ecec35b5ef" (UID: "02cdba52-fb35-42fa-91f2-77ecec35b5ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:41.811914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.811472 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.811914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.811487 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.811914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.811495 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.811914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.811504 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.811914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.811655 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-data" pod "02cdba52-fb35-42fa-91f2-77ecec35b5ef" (UID: "02cdba52-fb35-42fa-91f2-77ecec35b5ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:41.812942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.812918 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s"] Oct 30 06:20:41.814708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.814686 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-envoy" pod "02cdba52-fb35-42fa-91f2-77ecec35b5ef" (UID: "02cdba52-fb35-42fa-91f2-77ecec35b5ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:41.814780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.814709 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-token" pod "02cdba52-fb35-42fa-91f2-77ecec35b5ef" (UID: "02cdba52-fb35-42fa-91f2-77ecec35b5ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:41.814884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.814868 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-podinfo" pod "02cdba52-fb35-42fa-91f2-77ecec35b5ef" (UID: "02cdba52-fb35-42fa-91f2-77ecec35b5ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:41.816518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.816491 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-kube-api-access-bp5gj" pod "02cdba52-fb35-42fa-91f2-77ecec35b5ef" (UID: "02cdba52-fb35-42fa-91f2-77ecec35b5ef"). InnerVolumeSpecName "kube-api-access-bp5gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:41.817717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.817695 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-5f8747bcd7-q746s"] Oct 30 06:20:41.912829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.912787 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.912829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.912819 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.912829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.912833 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp5gj\" (UniqueName: \"kubernetes.io/projected/02cdba52-fb35-42fa-91f2-77ecec35b5ef-kube-api-access-bp5gj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.913048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.912847 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.913048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:41.912860 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02cdba52-fb35-42fa-91f2-77ecec35b5ef-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:42.118663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:42.118628 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a23f0c-7108-43e4-aa56-06005da64513" path="/var/lib/kubelet/pods/88a23f0c-7108-43e4-aa56-06005da64513/volumes" Oct 30 06:20:42.119129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:42.119108 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c9a551-5a64-4ff7-a465-5b0e29ba004c" path="/var/lib/kubelet/pods/b6c9a551-5a64-4ff7-a465-5b0e29ba004c/volumes" Oct 30 06:20:42.119612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:42.119592 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21ab9cb-4b51-467e-83d9-dbb0f9386660" path="/var/lib/kubelet/pods/f21ab9cb-4b51-467e-83d9-dbb0f9386660/volumes" Oct 30 06:20:42.350117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:42.350078 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:20:42.350279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:42.350171 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 06:20:42.431462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:42.431371 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" Oct 30 06:20:42.431462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:42.431392 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j" event={"ID":"02cdba52-fb35-42fa-91f2-77ecec35b5ef","Type":"ContainerDied","Data":"e9c3730d92cdd9ad9f3ac841c80be39201c7135b1a1ef22c1e1d0d265c0a5154"} Oct 30 06:20:42.431462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:42.431460 5001 scope.go:122] "RemoveContainer" containerID="da48dd99c1ee6e995240909fd08725313186c9afdd385858b5df069b79f2eb6a" Oct 30 06:20:42.495111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:42.495076 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j"] Oct 30 06:20:42.506733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:42.506693 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8272af-openshift-default-955c6877-zjm8j"] Oct 30 06:20:43.354376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:43.354344 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:20:43.354787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:43.354751 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:20:43.356141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:43.356122 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:20:44.119018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:44.118984 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" path="/var/lib/kubelet/pods/02cdba52-fb35-42fa-91f2-77ecec35b5ef/volumes" Oct 30 06:20:45.239624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.239593 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt"] Oct 30 06:20:45.240232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240217 5001 cpu_manager.go:414] "containerMap: removing container" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" Oct 30 06:20:45.240282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240233 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" Oct 30 06:20:45.240282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240258 5001 cpu_manager.go:414] "containerMap: removing container" podUID="88a23f0c-7108-43e4-aa56-06005da64513" containerName="payload-processing" Oct 30 06:20:45.240282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240263 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a23f0c-7108-43e4-aa56-06005da64513" containerName="payload-processing" Oct 30 06:20:45.240282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240273 5001 cpu_manager.go:414] "containerMap: removing container" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" Oct 30 06:20:45.240282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240278 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" Oct 30 06:20:45.240462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240288 5001 cpu_manager.go:414] "containerMap: removing container" podUID="93e99c8e-12fc-4412-9cb9-4612ad6869d9" containerName="revoke-keys" Oct 30 06:20:45.240462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240293 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e99c8e-12fc-4412-9cb9-4612ad6869d9" containerName="revoke-keys" Oct 30 06:20:45.240462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240330 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f21ab9cb-4b51-467e-83d9-dbb0f9386660" containerName="payload-pre-processing" Oct 30 06:20:45.240462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240335 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21ab9cb-4b51-467e-83d9-dbb0f9386660" containerName="payload-pre-processing" Oct 30 06:20:45.240462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240423 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" Oct 30 06:20:45.240462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240434 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a23f0c-7108-43e4-aa56-06005da64513" containerName="payload-processing" Oct 30 06:20:45.240462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240442 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cdba52-fb35-42fa-91f2-77ecec35b5ef" containerName="istio-proxy" Oct 30 06:20:45.240462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240449 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e99c8e-12fc-4412-9cb9-4612ad6869d9" containerName="revoke-keys" Oct 30 06:20:45.240462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.240457 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21ab9cb-4b51-467e-83d9-dbb0f9386660" containerName="payload-pre-processing" Oct 30 06:20:45.245549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.245525 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:20:45.255811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.255778 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt"] Oct 30 06:20:45.264711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.264677 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-520bc4\"" Oct 30 06:20:45.271344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.271298 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-520bc4-dockercfg-zsrst\"" Oct 30 06:20:45.359357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.359322 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc"] Oct 30 06:20:45.361477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.361433 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7cf442c-48ab-4054-8cb6-96680ce55215-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt\" (UID: \"d7cf442c-48ab-4054-8cb6-96680ce55215\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:20:45.361748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.361720 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf4s9\" (UniqueName: \"kubernetes.io/projected/d7cf442c-48ab-4054-8cb6-96680ce55215-kube-api-access-cf4s9\") pod \"payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt\" (UID: \"d7cf442c-48ab-4054-8cb6-96680ce55215\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:20:45.371921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.365154 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:20:45.418333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.415486 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc"] Oct 30 06:20:45.530344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.519360 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7cf442c-48ab-4054-8cb6-96680ce55215-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt\" (UID: \"d7cf442c-48ab-4054-8cb6-96680ce55215\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:20:45.556409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.555555 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfflk\" (UniqueName: \"kubernetes.io/projected/8d001f2c-4644-4fb5-a93f-0935613db130-kube-api-access-pfflk\") pod \"payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc\" (UID: \"8d001f2c-4644-4fb5-a93f-0935613db130\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:20:45.556409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.555740 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d001f2c-4644-4fb5-a93f-0935613db130-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc\" (UID: \"8d001f2c-4644-4fb5-a93f-0935613db130\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:20:45.556409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.555822 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4s9\" (UniqueName: \"kubernetes.io/projected/d7cf442c-48ab-4054-8cb6-96680ce55215-kube-api-access-cf4s9\") pod \"payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt\" (UID: \"d7cf442c-48ab-4054-8cb6-96680ce55215\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:20:45.556409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.533665 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7cf442c-48ab-4054-8cb6-96680ce55215-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt\" (UID: \"d7cf442c-48ab-4054-8cb6-96680ce55215\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:20:45.623333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.622680 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf4s9\" (UniqueName: \"kubernetes.io/projected/d7cf442c-48ab-4054-8cb6-96680ce55215-kube-api-access-cf4s9\") pod \"payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt\" (UID: \"d7cf442c-48ab-4054-8cb6-96680ce55215\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:20:45.656820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.656785 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d001f2c-4644-4fb5-a93f-0935613db130-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc\" (UID: \"8d001f2c-4644-4fb5-a93f-0935613db130\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:20:45.657048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.656961 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfflk\" (UniqueName: \"kubernetes.io/projected/8d001f2c-4644-4fb5-a93f-0935613db130-kube-api-access-pfflk\") pod \"payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc\" (UID: \"8d001f2c-4644-4fb5-a93f-0935613db130\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:20:45.657767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.657731 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d001f2c-4644-4fb5-a93f-0935613db130-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc\" (UID: \"8d001f2c-4644-4fb5-a93f-0935613db130\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:20:45.684200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.684161 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfflk\" (UniqueName: \"kubernetes.io/projected/8d001f2c-4644-4fb5-a93f-0935613db130-kube-api-access-pfflk\") pod \"payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc\" (UID: \"8d001f2c-4644-4fb5-a93f-0935613db130\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:20:45.826252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.826153 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2"] Oct 30 06:20:45.830703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.830673 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31407e-xk7s2"] Oct 30 06:20:45.866181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.866147 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:20:45.924524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:45.916959 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:20:46.122917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:46.122886 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="885d0da5-c3f7-49db-aa1f-b02567b3b89b" path="/var/lib/kubelet/pods/885d0da5-c3f7-49db-aa1f-b02567b3b89b/volumes" Oct 30 06:20:46.272834 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:46.272808 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt"] Oct 30 06:20:46.275807 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:20:46.275779 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7cf442c_48ab_4054_8cb6_96680ce55215.slice/crio-0906675c88488ead4c4f7b669ca266c3ce525deda383739ded6f4f8c8ae15d5c WatchSource:0}: Error finding container 0906675c88488ead4c4f7b669ca266c3ce525deda383739ded6f4f8c8ae15d5c: Status 404 returned error can't find the container with id 0906675c88488ead4c4f7b669ca266c3ce525deda383739ded6f4f8c8ae15d5c Oct 30 06:20:46.333596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:46.333565 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc"] Oct 30 06:20:46.334669 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:20:46.334645 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d001f2c_4644_4fb5_a93f_0935613db130.slice/crio-c36be0784cec3939c16c3196e67e242c558bc3e8a5b1c5975c39ac7eaea88bd9 WatchSource:0}: Error finding container c36be0784cec3939c16c3196e67e242c558bc3e8a5b1c5975c39ac7eaea88bd9: Status 404 returned error can't find the container with id c36be0784cec3939c16c3196e67e242c558bc3e8a5b1c5975c39ac7eaea88bd9 Oct 30 06:20:46.622722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:46.622674 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" event={"ID":"d7cf442c-48ab-4054-8cb6-96680ce55215","Type":"ContainerStarted","Data":"92a4c626829624caff00ce477c6b5209eeabfb76ba1f15e01a27404c30f8ba02"} Oct 30 06:20:46.622925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:46.622740 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" event={"ID":"d7cf442c-48ab-4054-8cb6-96680ce55215","Type":"ContainerStarted","Data":"0906675c88488ead4c4f7b669ca266c3ce525deda383739ded6f4f8c8ae15d5c"} Oct 30 06:20:46.622925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:46.622768 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:20:46.624682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:46.624657 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" event={"ID":"8d001f2c-4644-4fb5-a93f-0935613db130","Type":"ContainerStarted","Data":"dadad052b248145b33c3f118e685d1ed6f5ac7597d39b215bc12f87f287a4a1e"} Oct 30 06:20:46.624819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:46.624705 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" event={"ID":"8d001f2c-4644-4fb5-a93f-0935613db130","Type":"ContainerStarted","Data":"c36be0784cec3939c16c3196e67e242c558bc3e8a5b1c5975c39ac7eaea88bd9"} Oct 30 06:20:46.624819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:46.624775 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:20:46.667110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:46.667049 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" podStartSLOduration=1.667029872 podStartE2EDuration="1.667029872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:46.663584237 +0000 UTC m=+1660.659541247" watchObservedRunningTime="2026-10-30 06:20:46.667029872 +0000 UTC m=+1660.662986883" Oct 30 06:20:46.705355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:46.705287 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" podStartSLOduration=1.705272248 podStartE2EDuration="1.705272248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:46.704522107 +0000 UTC m=+1660.700479118" watchObservedRunningTime="2026-10-30 06:20:46.705272248 +0000 UTC m=+1660.701229258" Oct 30 06:20:48.796987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:48.796956 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m"] Oct 30 06:20:48.797505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:48.797214 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" containerID="cri-o://110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755" gracePeriod=30 Oct 30 06:20:49.503784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.503752 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h"] Oct 30 06:20:49.508164 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.508138 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:49.528678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.528650 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h"] Oct 30 06:20:49.534905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.534864 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:20:49.535066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.534955 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:20:49.603840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.603810 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:20:49.604015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.603855 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:20:49.630616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.630558 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:49.630844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.630819 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv98h\" (UniqueName: \"kubernetes.io/projected/705f3a7b-4d9d-454f-aced-b12297375238-kube-api-access-nv98h\") pod \"maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:49.630891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.630859 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:49.731805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.731772 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv98h\" (UniqueName: \"kubernetes.io/projected/705f3a7b-4d9d-454f-aced-b12297375238-kube-api-access-nv98h\") pod \"maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:49.731805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.731809 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:49.732074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.731876 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:49.732561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.732539 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:49.734659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.734631 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:49.754849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.754770 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv98h\" (UniqueName: \"kubernetes.io/projected/705f3a7b-4d9d-454f-aced-b12297375238-kube-api-access-nv98h\") pod \"maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:49.823197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:49.823163 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:50.139115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:50.139090 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h"] Oct 30 06:20:50.141221 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:20:50.141191 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705f3a7b_4d9d_454f_aced_b12297375238.slice/crio-457e27f38edd1056dbd535e5f23389bda591dfe68d7984a89a7590e6ed090c03 WatchSource:0}: Error finding container 457e27f38edd1056dbd535e5f23389bda591dfe68d7984a89a7590e6ed090c03: Status 404 returned error can't find the container with id 457e27f38edd1056dbd535e5f23389bda591dfe68d7984a89a7590e6ed090c03 Oct 30 06:20:50.672064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:50.672025 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" event={"ID":"705f3a7b-4d9d-454f-aced-b12297375238","Type":"ContainerStarted","Data":"0620690edad1eee9601fdb79c8db4c1f268557c745585737c1548df79152e85d"} Oct 30 06:20:50.672257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:50.672121 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" event={"ID":"705f3a7b-4d9d-454f-aced-b12297375238","Type":"ContainerStarted","Data":"457e27f38edd1056dbd535e5f23389bda591dfe68d7984a89a7590e6ed090c03"} Oct 30 06:20:50.712693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:50.712544 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" podStartSLOduration=1.712525562 podStartE2EDuration="1.712525562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:50.711117258 +0000 UTC m=+1664.707074268" watchObservedRunningTime="2026-10-30 06:20:50.712525562 +0000 UTC m=+1664.708482573" Oct 30 06:20:51.684669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:51.684619 5001 generic.go:356] "Generic (PLEG): container finished" podID="705f3a7b-4d9d-454f-aced-b12297375238" containerID="0620690edad1eee9601fdb79c8db4c1f268557c745585737c1548df79152e85d" exitCode=0 Oct 30 06:20:51.684669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:51.684676 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" event={"ID":"705f3a7b-4d9d-454f-aced-b12297375238","Type":"ContainerDied","Data":"0620690edad1eee9601fdb79c8db4c1f268557c745585737c1548df79152e85d"} Oct 30 06:20:52.864883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:52.864855 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:52.928217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:52.928171 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705f3a7b-4d9d-454f-aced-b12297375238-kube-api-access-nv98h\" (UniqueName: \"kubernetes.io/projected/705f3a7b-4d9d-454f-aced-b12297375238-kube-api-access-nv98h\") pod \"705f3a7b-4d9d-454f-aced-b12297375238\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " Oct 30 06:20:52.928436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:52.928245 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-openshift-ca\") pod \"705f3a7b-4d9d-454f-aced-b12297375238\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " Oct 30 06:20:52.928436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:52.928279 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-xks-ca\") pod \"705f3a7b-4d9d-454f-aced-b12297375238\" (UID: \"705f3a7b-4d9d-454f-aced-b12297375238\") " Oct 30 06:20:52.930066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:52.930035 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-openshift-ca" pod "705f3a7b-4d9d-454f-aced-b12297375238" (UID: "705f3a7b-4d9d-454f-aced-b12297375238"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:52.931097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:52.931063 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-xks-ca" pod "705f3a7b-4d9d-454f-aced-b12297375238" (UID: "705f3a7b-4d9d-454f-aced-b12297375238"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:52.931270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:52.931151 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705f3a7b-4d9d-454f-aced-b12297375238-kube-api-access-nv98h" pod "705f3a7b-4d9d-454f-aced-b12297375238" (UID: "705f3a7b-4d9d-454f-aced-b12297375238"). InnerVolumeSpecName "kube-api-access-nv98h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:53.030585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:53.030494 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv98h\" (UniqueName: \"kubernetes.io/projected/705f3a7b-4d9d-454f-aced-b12297375238-kube-api-access-nv98h\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:53.030585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:53.030526 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:53.030585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:53.030536 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705f3a7b-4d9d-454f-aced-b12297375238-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:53.705458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:53.705429 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" Oct 30 06:20:53.705458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:53.705452 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h" event={"ID":"705f3a7b-4d9d-454f-aced-b12297375238","Type":"ContainerDied","Data":"457e27f38edd1056dbd535e5f23389bda591dfe68d7984a89a7590e6ed090c03"} Oct 30 06:20:53.705688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:53.705501 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="457e27f38edd1056dbd535e5f23389bda591dfe68d7984a89a7590e6ed090c03" Oct 30 06:20:54.220468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.220444 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:20:54.273763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.273726 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-kube-api-access-m9m74\" (UniqueName: \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-kube-api-access-m9m74\") pod \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " Oct 30 06:20:54.273763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.273769 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-data\") pod \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " Oct 30 06:20:54.274039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.273800 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-token\" (UniqueName: \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-token\") pod \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " Oct 30 06:20:54.274039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.273879 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-credential-socket\") pod \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " Oct 30 06:20:54.274039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.273986 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-certs\") pod \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " Oct 30 06:20:54.274203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.274051 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-envoy\") pod \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " Oct 30 06:20:54.274203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.274082 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-data" pod "f072d364-6b3a-4b8b-9ae8-b9ff16503cda" (UID: "f072d364-6b3a-4b8b-9ae8-b9ff16503cda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:54.274203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.274108 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istiod-ca-cert\") pod \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " Oct 30 06:20:54.274203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.274142 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-podinfo\") pod \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " Oct 30 06:20:54.274203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.274172 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-credential-socket" pod "f072d364-6b3a-4b8b-9ae8-b9ff16503cda" (UID: "f072d364-6b3a-4b8b-9ae8-b9ff16503cda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:54.274568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.274239 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-socket\") pod \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\" (UID: \"f072d364-6b3a-4b8b-9ae8-b9ff16503cda\") " Oct 30 06:20:54.274568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.274296 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-certs" pod "f072d364-6b3a-4b8b-9ae8-b9ff16503cda" (UID: "f072d364-6b3a-4b8b-9ae8-b9ff16503cda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:54.274568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.274537 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-socket" pod "f072d364-6b3a-4b8b-9ae8-b9ff16503cda" (UID: "f072d364-6b3a-4b8b-9ae8-b9ff16503cda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:54.275359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.275039 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istiod-ca-cert" pod "f072d364-6b3a-4b8b-9ae8-b9ff16503cda" (UID: "f072d364-6b3a-4b8b-9ae8-b9ff16503cda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:54.275508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.275421 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.275508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.275445 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.275508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.275460 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.275508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.275473 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.275508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.275487 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.277073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.277013 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-token" pod "f072d364-6b3a-4b8b-9ae8-b9ff16503cda" (UID: "f072d364-6b3a-4b8b-9ae8-b9ff16503cda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:54.277235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.277223 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-kube-api-access-m9m74" pod "f072d364-6b3a-4b8b-9ae8-b9ff16503cda" (UID: "f072d364-6b3a-4b8b-9ae8-b9ff16503cda"). InnerVolumeSpecName "kube-api-access-m9m74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:54.277699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.277651 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-podinfo" pod "f072d364-6b3a-4b8b-9ae8-b9ff16503cda" (UID: "f072d364-6b3a-4b8b-9ae8-b9ff16503cda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:54.278164 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.278131 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-envoy" pod "f072d364-6b3a-4b8b-9ae8-b9ff16503cda" (UID: "f072d364-6b3a-4b8b-9ae8-b9ff16503cda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:54.377236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.377206 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.377236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.377235 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.377467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.377250 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9m74\" (UniqueName: \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-kube-api-access-m9m74\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.377467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.377262 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f072d364-6b3a-4b8b-9ae8-b9ff16503cda-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.407091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.407040 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh"] Oct 30 06:20:54.407651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.407594 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" containerID="cri-o://9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21" gracePeriod=30 Oct 30 06:20:54.716391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.716358 5001 generic.go:356] "Generic (PLEG): container finished" podID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerID="110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755" exitCode=0 Oct 30 06:20:54.716574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.716416 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" Oct 30 06:20:54.716574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.716446 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" event={"ID":"f072d364-6b3a-4b8b-9ae8-b9ff16503cda","Type":"ContainerDied","Data":"110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755"} Oct 30 06:20:54.716574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.716504 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m" event={"ID":"f072d364-6b3a-4b8b-9ae8-b9ff16503cda","Type":"ContainerDied","Data":"24f06b7317d1369451bf823bdaa810e64d71bee9db15b41d405b772eff888cb9"} Oct 30 06:20:54.716574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.716536 5001 scope.go:122] "RemoveContainer" containerID="110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755" Oct 30 06:20:54.729977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.729954 5001 scope.go:122] "RemoveContainer" containerID="8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa" Oct 30 06:20:54.742114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.742091 5001 scope.go:122] "RemoveContainer" containerID="110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755" Oct 30 06:20:54.742499 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:54.742477 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755\": container with ID starting with 110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755 not found: ID does not exist" containerID="110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755" Oct 30 06:20:54.742574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.742509 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755"} err="failed to get container status \"110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755\": rpc error: code = NotFound desc = could not find container \"110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755\": container with ID starting with 110c935b9a7d533cd49755bf43e0a6968c8989e456b67df7e9e4018bb6c14755 not found: ID does not exist" Oct 30 06:20:54.742574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.742528 5001 scope.go:122] "RemoveContainer" containerID="8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa" Oct 30 06:20:54.742854 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:54.742826 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa\": container with ID starting with 8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa not found: ID does not exist" containerID="8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa" Oct 30 06:20:54.742959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.742857 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa"} err="failed to get container status \"8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa\": rpc error: code = NotFound desc = could not find container \"8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa\": container with ID starting with 8ee7464cd4f83fce76751d3af43e56fac95412687c13168e791bab9f8dfbdffa not found: ID does not exist" Oct 30 06:20:54.783333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.782652 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m"] Oct 30 06:20:54.796028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:54.795990 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f50f6e-openshift-default-fcbf8bcd7-dm45m"] Oct 30 06:20:55.533342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:55.533272 5001 patch_prober.go:29] interesting pod/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:20:55.533753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:55.533342 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:20:56.118129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:56.118098 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" path="/var/lib/kubelet/pods/f072d364-6b3a-4b8b-9ae8-b9ff16503cda/volumes" Oct 30 06:20:57.048297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.048268 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn"] Oct 30 06:20:57.049000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.048982 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" Oct 30 06:20:57.049000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.049002 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" Oct 30 06:20:57.049134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.049052 5001 cpu_manager.go:414] "containerMap: removing container" podUID="705f3a7b-4d9d-454f-aced-b12297375238" containerName="revoke-keys" Oct 30 06:20:57.049134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.049061 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705f3a7b-4d9d-454f-aced-b12297375238" containerName="revoke-keys" Oct 30 06:20:57.049244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.049194 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" Oct 30 06:20:57.049244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.049208 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="705f3a7b-4d9d-454f-aced-b12297375238" containerName="revoke-keys" Oct 30 06:20:57.061235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.061204 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.080331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.080285 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn"] Oct 30 06:20:57.084948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.084923 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-bbd0e5-openshift-default-dockercfg-bmw9c\"" Oct 30 06:20:57.125848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.125806 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655aebda-9cbb-4aae-b560-ee17928630b9-istio-podinfo\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.126059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.125875 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655aebda-9cbb-4aae-b560-ee17928630b9-istiod-ca-cert\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.126059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.125991 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l6nk\" (UniqueName: \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-kube-api-access-8l6nk\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.126059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.126054 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-certs\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.126226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.126204 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-istio-token\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.126280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.126246 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-socket\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.126412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.126387 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-data\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.126494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.126479 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-envoy\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.126599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.126585 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-credential-socket\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.242286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.242238 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-certs\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.244487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.244460 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-istio-token\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.245012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.244992 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-socket\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.245207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.245190 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-data\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.245441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.245418 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-envoy\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.245605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.245586 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-credential-socket\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.247012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.246993 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655aebda-9cbb-4aae-b560-ee17928630b9-istio-podinfo\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.247187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.247172 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655aebda-9cbb-4aae-b560-ee17928630b9-istiod-ca-cert\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.247345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.247328 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6nk\" (UniqueName: \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-kube-api-access-8l6nk\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.251332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.249972 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-credential-socket\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.251332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.244348 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-certs\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.251332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.250601 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-socket\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.251332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.246591 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-data\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.260705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.251716 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655aebda-9cbb-4aae-b560-ee17928630b9-istiod-ca-cert\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.298714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.298597 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655aebda-9cbb-4aae-b560-ee17928630b9-istio-podinfo\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.299346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.299322 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-envoy\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.302118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.302096 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6nk\" (UniqueName: \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-kube-api-access-8l6nk\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.303471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.303445 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-istio-token\") pod \"e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.392618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.392584 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:57.635657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.635630 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:20:57.635824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.635711 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:20:57.795730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.795696 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn"] Oct 30 06:20:57.797632 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:20:57.797606 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655aebda_9cbb_4aae_b560_ee17928630b9.slice/crio-355a81bcc1c0eaa45a5dad4e9abbc7be8b3171852b9fc02492ba9d4c50bb05fa WatchSource:0}: Error finding container 355a81bcc1c0eaa45a5dad4e9abbc7be8b3171852b9fc02492ba9d4c50bb05fa: Status 404 returned error can't find the container with id 355a81bcc1c0eaa45a5dad4e9abbc7be8b3171852b9fc02492ba9d4c50bb05fa Oct 30 06:20:57.800581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.800546 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:57.800669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.800612 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:57.800669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:57.800641 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:58.758612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:58.758574 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" event={"ID":"655aebda-9cbb-4aae-b560-ee17928630b9","Type":"ContainerStarted","Data":"7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8"} Oct 30 06:20:58.759064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:58.758641 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" event={"ID":"655aebda-9cbb-4aae-b560-ee17928630b9","Type":"ContainerStarted","Data":"355a81bcc1c0eaa45a5dad4e9abbc7be8b3171852b9fc02492ba9d4c50bb05fa"} Oct 30 06:20:58.828947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:58.828882 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" podStartSLOduration=1.828863534 podStartE2EDuration="1.828863534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:58.816520969 +0000 UTC m=+1672.812477980" watchObservedRunningTime="2026-10-30 06:20:58.828863534 +0000 UTC m=+1672.824820545" Oct 30 06:20:59.393862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.393817 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:59.399801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.399773 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:59.778207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.778177 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:20:59.778723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.778469 5001 generic.go:356] "Generic (PLEG): container finished" podID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerID="9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21" exitCode=0 Oct 30 06:20:59.778723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.778597 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" event={"ID":"286fb2e3-76ff-44f6-aaf4-c537b0673ce5","Type":"ContainerDied","Data":"9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21"} Oct 30 06:20:59.778723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.778654 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" event={"ID":"286fb2e3-76ff-44f6-aaf4-c537b0673ce5","Type":"ContainerDied","Data":"defcea1e9c5975958bc9c0cd14b96bbaa3b404d897b5758814796a979043c3f1"} Oct 30 06:20:59.778723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.778685 5001 scope.go:122] "RemoveContainer" containerID="9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21" Oct 30 06:20:59.779132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.779115 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:59.780262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.780210 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:20:59.794753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.794735 5001 scope.go:122] "RemoveContainer" containerID="18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30" Oct 30 06:20:59.807531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.807512 5001 scope.go:122] "RemoveContainer" containerID="9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21" Oct 30 06:20:59.807866 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:59.807845 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21\": container with ID starting with 9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21 not found: ID does not exist" containerID="9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21" Oct 30 06:20:59.807926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.807875 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21"} err="failed to get container status \"9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21\": rpc error: code = NotFound desc = could not find container \"9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21\": container with ID starting with 9730f1cb8ea626567961d492f170d9c079a6aca5f352128108a4c3978578ee21 not found: ID does not exist" Oct 30 06:20:59.807926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.807895 5001 scope.go:122] "RemoveContainer" containerID="18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30" Oct 30 06:20:59.808179 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:20:59.808164 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30\": container with ID starting with 18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30 not found: ID does not exist" containerID="18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30" Oct 30 06:20:59.808218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.808192 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30"} err="failed to get container status \"18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30\": rpc error: code = NotFound desc = could not find container \"18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30\": container with ID starting with 18b64bada648d7abf24a72a331bc70a3daa060f24fb0443fc028af1e6b757b30 not found: ID does not exist" Oct 30 06:20:59.928591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.928548 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-credential-socket\") pod \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " Oct 30 06:20:59.928591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.928588 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-kube-api-access-q7cgw\" (UniqueName: \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-kube-api-access-q7cgw\") pod \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " Oct 30 06:20:59.928859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.928610 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-socket\") pod \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " Oct 30 06:20:59.928859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.928740 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-token\" (UniqueName: \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-token\") pod \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " Oct 30 06:20:59.929001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.928966 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-credential-socket" pod "286fb2e3-76ff-44f6-aaf4-c537b0673ce5" (UID: "286fb2e3-76ff-44f6-aaf4-c537b0673ce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:59.929001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.928989 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-socket" pod "286fb2e3-76ff-44f6-aaf4-c537b0673ce5" (UID: "286fb2e3-76ff-44f6-aaf4-c537b0673ce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:59.929181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.929162 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istiod-ca-cert\") pod \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " Oct 30 06:20:59.929247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.929214 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-certs\") pod \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " Oct 30 06:20:59.929322 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.929278 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-podinfo\") pod \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " Oct 30 06:20:59.930518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.929481 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-envoy\") pod \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " Oct 30 06:20:59.930518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.929528 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-data\") pod \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\" (UID: \"286fb2e3-76ff-44f6-aaf4-c537b0673ce5\") " Oct 30 06:20:59.930518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.929639 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-certs" pod "286fb2e3-76ff-44f6-aaf4-c537b0673ce5" (UID: "286fb2e3-76ff-44f6-aaf4-c537b0673ce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:59.930518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.929973 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-data" pod "286fb2e3-76ff-44f6-aaf4-c537b0673ce5" (UID: "286fb2e3-76ff-44f6-aaf4-c537b0673ce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:59.930518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.930273 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:59.930518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.930294 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:59.930518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.930331 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:59.930518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.930347 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:20:59.931122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.931091 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istiod-ca-cert" pod "286fb2e3-76ff-44f6-aaf4-c537b0673ce5" (UID: "286fb2e3-76ff-44f6-aaf4-c537b0673ce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:59.933108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.933018 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-kube-api-access-q7cgw" pod "286fb2e3-76ff-44f6-aaf4-c537b0673ce5" (UID: "286fb2e3-76ff-44f6-aaf4-c537b0673ce5"). InnerVolumeSpecName "kube-api-access-q7cgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:59.933396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.933376 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-token" pod "286fb2e3-76ff-44f6-aaf4-c537b0673ce5" (UID: "286fb2e3-76ff-44f6-aaf4-c537b0673ce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:59.934402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.934380 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-envoy" pod "286fb2e3-76ff-44f6-aaf4-c537b0673ce5" (UID: "286fb2e3-76ff-44f6-aaf4-c537b0673ce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:59.934556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:20:59.934534 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-podinfo" pod "286fb2e3-76ff-44f6-aaf4-c537b0673ce5" (UID: "286fb2e3-76ff-44f6-aaf4-c537b0673ce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:00.031977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:00.031884 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.031977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:00.031926 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.031977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:00.031938 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7cgw\" (UniqueName: \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-kube-api-access-q7cgw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.031977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:00.031949 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.031977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:00.031957 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/286fb2e3-76ff-44f6-aaf4-c537b0673ce5-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.788889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:00.788851 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh" Oct 30 06:21:00.854540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:00.854508 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh"] Oct 30 06:21:00.866258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:00.866219 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-31407e-openshift-default-6bf5cb57c5-7qnsh"] Oct 30 06:21:02.121191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:02.121148 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" path="/var/lib/kubelet/pods/286fb2e3-76ff-44f6-aaf4-c537b0673ce5/volumes" Oct 30 06:21:03.220810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.220768 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k"] Oct 30 06:21:03.221480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.221462 5001 cpu_manager.go:414] "containerMap: removing container" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" Oct 30 06:21:03.221536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.221481 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" Oct 30 06:21:03.221536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.221507 5001 cpu_manager.go:414] "containerMap: removing container" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" Oct 30 06:21:03.221536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.221512 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" Oct 30 06:21:03.221633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.221537 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" Oct 30 06:21:03.221633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.221543 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" Oct 30 06:21:03.221710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.221648 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" Oct 30 06:21:03.221710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.221659 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f072d364-6b3a-4b8b-9ae8-b9ff16503cda" containerName="istio-proxy" Oct 30 06:21:03.221710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.221666 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="286fb2e3-76ff-44f6-aaf4-c537b0673ce5" containerName="istio-proxy" Oct 30 06:21:03.227043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.227020 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:21:03.247381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.247352 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k"] Oct 30 06:21:03.253922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.253879 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-bbd0e5\"" Oct 30 06:21:03.274837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.274802 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl"] Oct 30 06:21:03.279609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.279574 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:21:03.297866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.297817 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-bbd0e5-dockercfg-qr2tc\"" Oct 30 06:21:03.299211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.299185 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl"] Oct 30 06:21:03.435869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.400462 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xnck\" (UniqueName: \"kubernetes.io/projected/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-kube-api-access-4xnck\") pod \"payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k\" (UID: \"a2ec7a66-b879-4514-9ba2-3249d56ae8ad\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:21:03.485390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.485235 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tgq\" (UniqueName: \"kubernetes.io/projected/d4324bad-9424-4246-8b75-5e6e2a208739-kube-api-access-99tgq\") pod \"payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl\" (UID: \"d4324bad-9424-4246-8b75-5e6e2a208739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:21:03.485853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.485832 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4324bad-9424-4246-8b75-5e6e2a208739-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl\" (UID: \"d4324bad-9424-4246-8b75-5e6e2a208739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:21:03.486212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.486194 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k\" (UID: \"a2ec7a66-b879-4514-9ba2-3249d56ae8ad\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:21:03.588030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.587995 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k\" (UID: \"a2ec7a66-b879-4514-9ba2-3249d56ae8ad\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:21:03.588213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.588170 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnck\" (UniqueName: \"kubernetes.io/projected/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-kube-api-access-4xnck\") pod \"payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k\" (UID: \"a2ec7a66-b879-4514-9ba2-3249d56ae8ad\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:21:03.588213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.588192 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99tgq\" (UniqueName: \"kubernetes.io/projected/d4324bad-9424-4246-8b75-5e6e2a208739-kube-api-access-99tgq\") pod \"payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl\" (UID: \"d4324bad-9424-4246-8b75-5e6e2a208739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:21:03.588367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.588223 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4324bad-9424-4246-8b75-5e6e2a208739-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl\" (UID: \"d4324bad-9424-4246-8b75-5e6e2a208739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:21:03.588940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.588913 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k\" (UID: \"a2ec7a66-b879-4514-9ba2-3249d56ae8ad\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:21:03.589054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.589038 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4324bad-9424-4246-8b75-5e6e2a208739-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl\" (UID: \"d4324bad-9424-4246-8b75-5e6e2a208739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:21:03.627989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.627956 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tgq\" (UniqueName: \"kubernetes.io/projected/d4324bad-9424-4246-8b75-5e6e2a208739-kube-api-access-99tgq\") pod \"payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl\" (UID: \"d4324bad-9424-4246-8b75-5e6e2a208739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:21:03.630747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.630723 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xnck\" (UniqueName: \"kubernetes.io/projected/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-kube-api-access-4xnck\") pod \"payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k\" (UID: \"a2ec7a66-b879-4514-9ba2-3249d56ae8ad\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:21:03.841764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.841668 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:21:03.898380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:03.898298 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:21:04.151471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:04.151439 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k"] Oct 30 06:21:04.152746 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:21:04.152713 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ec7a66_b879_4514_9ba2_3249d56ae8ad.slice/crio-0f349d2f3137db6b783da3fa4b3a2b97b3c027826f3a440b0e9088e5877d1ebe WatchSource:0}: Error finding container 0f349d2f3137db6b783da3fa4b3a2b97b3c027826f3a440b0e9088e5877d1ebe: Status 404 returned error can't find the container with id 0f349d2f3137db6b783da3fa4b3a2b97b3c027826f3a440b0e9088e5877d1ebe Oct 30 06:21:04.226997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:04.226972 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl"] Oct 30 06:21:04.229466 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:21:04.229421 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4324bad_9424_4246_8b75_5e6e2a208739.slice/crio-c451f187df51fd178eee4089dad4d28768bdcdb06c6a1ea9c55c93f4f9463c52 WatchSource:0}: Error finding container c451f187df51fd178eee4089dad4d28768bdcdb06c6a1ea9c55c93f4f9463c52: Status 404 returned error can't find the container with id c451f187df51fd178eee4089dad4d28768bdcdb06c6a1ea9c55c93f4f9463c52 Oct 30 06:21:04.826956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:04.826905 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" event={"ID":"a2ec7a66-b879-4514-9ba2-3249d56ae8ad","Type":"ContainerStarted","Data":"889bac59709ef3db1ba42b094eea07ccab5b54bdd565b78f8b24da9288278793"} Oct 30 06:21:04.827168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:04.826969 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" event={"ID":"a2ec7a66-b879-4514-9ba2-3249d56ae8ad","Type":"ContainerStarted","Data":"0f349d2f3137db6b783da3fa4b3a2b97b3c027826f3a440b0e9088e5877d1ebe"} Oct 30 06:21:04.827168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:04.826996 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:21:04.829054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:04.829022 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" event={"ID":"d4324bad-9424-4246-8b75-5e6e2a208739","Type":"ContainerStarted","Data":"b04ea33c7da34d5134f301448dbc09ea0c8b4772b4b9972f36e979abb2d571c9"} Oct 30 06:21:04.829219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:04.829082 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" event={"ID":"d4324bad-9424-4246-8b75-5e6e2a208739","Type":"ContainerStarted","Data":"c451f187df51fd178eee4089dad4d28768bdcdb06c6a1ea9c55c93f4f9463c52"} Oct 30 06:21:04.829219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:04.829158 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:21:04.870600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:04.870535 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" podStartSLOduration=1.870515312 podStartE2EDuration="1.870515312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:04.868791851 +0000 UTC m=+1678.864748869" watchObservedRunningTime="2026-10-30 06:21:04.870515312 +0000 UTC m=+1678.866472323" Oct 30 06:21:04.913028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:04.912977 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" podStartSLOduration=1.912961522 podStartE2EDuration="1.912961522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:04.911109995 +0000 UTC m=+1678.907066995" watchObservedRunningTime="2026-10-30 06:21:04.912961522 +0000 UTC m=+1678.908918537" Oct 30 06:21:07.976110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:07.976074 5001 scope.go:122] "RemoveContainer" containerID="524ec6ab7df0bec5d66f81bbd6d15e058293e68956dee798548fb0e4dc0353f7" Oct 30 06:21:07.987482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:07.987451 5001 scope.go:122] "RemoveContainer" containerID="9ba901a8870bb6b63fe5840a9431d5d417034c55fccdb4f34d9b73ee76d5402c" Oct 30 06:21:07.998104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:07.998076 5001 scope.go:122] "RemoveContainer" containerID="54d3ffbf5ab0fad768ecb225e95b3e2e3e50e71b0349b8094f1e4db092cab05c" Oct 30 06:21:08.008396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:08.008376 5001 scope.go:122] "RemoveContainer" containerID="f87f5f263bbbbdae80e5573f320b36928a09141a72b54830ac14fbbf130e1479" Oct 30 06:21:08.270655 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:21:08.270562 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:21:09.513803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:09.513761 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l"] Oct 30 06:21:09.528983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:09.528952 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9"] Oct 30 06:21:09.529191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:09.529167 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" podUID="5e5a1389-012c-4d7e-968e-b51a8bb781d6" containerName="payload-pre-processing" containerID="cri-o://af91c76d7597b8ebf3303e37f1617a1827e7a5666cffd5a8e02c87fec30961b3" gracePeriod=30 Oct 30 06:21:09.986331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:09.981513 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" event={"ID":"5e5a1389-012c-4d7e-968e-b51a8bb781d6","Type":"ContainerDied","Data":"af91c76d7597b8ebf3303e37f1617a1827e7a5666cffd5a8e02c87fec30961b3"} Oct 30 06:21:09.986331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:09.981459 5001 generic.go:356] "Generic (PLEG): container finished" podID="5e5a1389-012c-4d7e-968e-b51a8bb781d6" containerID="af91c76d7597b8ebf3303e37f1617a1827e7a5666cffd5a8e02c87fec30961b3" exitCode=0 Oct 30 06:21:09.986331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:09.981706 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" event={"ID":"5e5a1389-012c-4d7e-968e-b51a8bb781d6","Type":"ContainerDied","Data":"52e8011513290362a68a8dfe58391e3ec844e06ab49dc2d0a51f72421561d836"} Oct 30 06:21:09.986331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:09.981742 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52e8011513290362a68a8dfe58391e3ec844e06ab49dc2d0a51f72421561d836" Oct 30 06:21:09.986331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:09.982657 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" podUID="af3b8f19-f99e-4299-a4bf-bd9cbabafc49" containerName="payload-processing" containerID="cri-o://a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846" gracePeriod=30 Oct 30 06:21:09.986331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:09.983669 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:21:10.112330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.111451 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5a1389-012c-4d7e-968e-b51a8bb781d6-kube-api-access-rmks8\" (UniqueName: \"kubernetes.io/projected/5e5a1389-012c-4d7e-968e-b51a8bb781d6-kube-api-access-rmks8\") pod \"5e5a1389-012c-4d7e-968e-b51a8bb781d6\" (UID: \"5e5a1389-012c-4d7e-968e-b51a8bb781d6\") " Oct 30 06:21:10.112330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.112037 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5a1389-012c-4d7e-968e-b51a8bb781d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5a1389-012c-4d7e-968e-b51a8bb781d6-plugins-config-volume\") pod \"5e5a1389-012c-4d7e-968e-b51a8bb781d6\" (UID: \"5e5a1389-012c-4d7e-968e-b51a8bb781d6\") " Oct 30 06:21:10.112581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.112490 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5a1389-012c-4d7e-968e-b51a8bb781d6-plugins-config-volume" pod "5e5a1389-012c-4d7e-968e-b51a8bb781d6" (UID: "5e5a1389-012c-4d7e-968e-b51a8bb781d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:10.118170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.113022 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5a1389-012c-4d7e-968e-b51a8bb781d6-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:10.119084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.119045 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5a1389-012c-4d7e-968e-b51a8bb781d6-kube-api-access-rmks8" pod "5e5a1389-012c-4d7e-968e-b51a8bb781d6" (UID: "5e5a1389-012c-4d7e-968e-b51a8bb781d6"). InnerVolumeSpecName "kube-api-access-rmks8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:10.172941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.172894 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8"] Oct 30 06:21:10.173761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.173739 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a55a4b-nxdj8"] Oct 30 06:21:10.270264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.233963 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmks8\" (UniqueName: \"kubernetes.io/projected/5e5a1389-012c-4d7e-968e-b51a8bb781d6-kube-api-access-rmks8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:10.341886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.341854 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:21:10.435719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.435624 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-kube-api-access-glffx\" (UniqueName: \"kubernetes.io/projected/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-kube-api-access-glffx\") pod \"af3b8f19-f99e-4299-a4bf-bd9cbabafc49\" (UID: \"af3b8f19-f99e-4299-a4bf-bd9cbabafc49\") " Oct 30 06:21:10.435719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.435693 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-plugins-config-volume\") pod \"af3b8f19-f99e-4299-a4bf-bd9cbabafc49\" (UID: \"af3b8f19-f99e-4299-a4bf-bd9cbabafc49\") " Oct 30 06:21:10.436281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.436235 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-plugins-config-volume" pod "af3b8f19-f99e-4299-a4bf-bd9cbabafc49" (UID: "af3b8f19-f99e-4299-a4bf-bd9cbabafc49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:10.436508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.436485 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:10.438037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.438008 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-kube-api-access-glffx" pod "af3b8f19-f99e-4299-a4bf-bd9cbabafc49" (UID: "af3b8f19-f99e-4299-a4bf-bd9cbabafc49"). InnerVolumeSpecName "kube-api-access-glffx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:10.537973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.537931 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glffx\" (UniqueName: \"kubernetes.io/projected/af3b8f19-f99e-4299-a4bf-bd9cbabafc49-kube-api-access-glffx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:10.992761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.992731 5001 generic.go:356] "Generic (PLEG): container finished" podID="af3b8f19-f99e-4299-a4bf-bd9cbabafc49" containerID="a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846" exitCode=0 Oct 30 06:21:10.992973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.992792 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" Oct 30 06:21:10.992973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.992818 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" event={"ID":"af3b8f19-f99e-4299-a4bf-bd9cbabafc49","Type":"ContainerDied","Data":"a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846"} Oct 30 06:21:10.992973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.992873 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l" event={"ID":"af3b8f19-f99e-4299-a4bf-bd9cbabafc49","Type":"ContainerDied","Data":"1c0c9621223ac861061a4a31710d376fe696936c097f4fd569a3e70353d51617"} Oct 30 06:21:10.992973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.992892 5001 scope.go:122] "RemoveContainer" containerID="a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846" Oct 30 06:21:10.993184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:10.993040 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9" Oct 30 06:21:11.007699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:11.007680 5001 scope.go:122] "RemoveContainer" containerID="a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846" Oct 30 06:21:11.008020 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:21:11.007999 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846\": container with ID starting with a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846 not found: ID does not exist" containerID="a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846" Oct 30 06:21:11.008109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:11.008027 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846"} err="failed to get container status \"a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846\": rpc error: code = NotFound desc = could not find container \"a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846\": container with ID starting with a230fdd83d269a9e070665b55531c7fdd9fc055d3e54cf23aba423f2840db846 not found: ID does not exist" Oct 30 06:21:11.045721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:11.045693 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9"] Oct 30 06:21:11.056676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:11.056646 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a55a4b-846f47dc79-nr2z9"] Oct 30 06:21:11.091795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:11.091763 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l"] Oct 30 06:21:11.095777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:11.095754 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a55a4b-b845dbd76-dgs6l"] Oct 30 06:21:12.118542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:12.118507 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5a1389-012c-4d7e-968e-b51a8bb781d6" path="/var/lib/kubelet/pods/5e5a1389-012c-4d7e-968e-b51a8bb781d6/volumes" Oct 30 06:21:12.118941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:12.118927 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e99c8e-12fc-4412-9cb9-4612ad6869d9" path="/var/lib/kubelet/pods/93e99c8e-12fc-4412-9cb9-4612ad6869d9/volumes" Oct 30 06:21:12.119272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:12.119259 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3b8f19-f99e-4299-a4bf-bd9cbabafc49" path="/var/lib/kubelet/pods/af3b8f19-f99e-4299-a4bf-bd9cbabafc49/volumes" Oct 30 06:21:13.036211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:13.036173 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6"] Oct 30 06:21:13.047460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:13.047432 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba69e6d-xz6t6"] Oct 30 06:21:13.338663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:13.338586 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:21:14.119266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:14.119229 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf05d5f-d505-4701-8381-ace1143a93b2" path="/var/lib/kubelet/pods/0cf05d5f-d505-4701-8381-ace1143a93b2/volumes" Oct 30 06:21:15.856253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:15.856223 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:21:15.856866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:15.856327 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:21:18.057466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:18.057349 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b"] Oct 30 06:21:18.061602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:18.061562 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" containerID="cri-o://fb9cc9241688c255e9cbb8ac6c0778f463d76e76adbbc82977642a2b6d76f651" gracePeriod=30 Oct 30 06:21:18.163950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:18.159358 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:21:18.163950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:18.159434 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:21:20.322256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.322196 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4"] Oct 30 06:21:20.323213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.323189 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5e5a1389-012c-4d7e-968e-b51a8bb781d6" containerName="payload-pre-processing" Oct 30 06:21:20.323293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.323217 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5a1389-012c-4d7e-968e-b51a8bb781d6" containerName="payload-pre-processing" Oct 30 06:21:20.323293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.323255 5001 cpu_manager.go:414] "containerMap: removing container" podUID="af3b8f19-f99e-4299-a4bf-bd9cbabafc49" containerName="payload-processing" Oct 30 06:21:20.323293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.323263 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3b8f19-f99e-4299-a4bf-bd9cbabafc49" containerName="payload-processing" Oct 30 06:21:20.323500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.323478 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5a1389-012c-4d7e-968e-b51a8bb781d6" containerName="payload-pre-processing" Oct 30 06:21:20.323500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.323496 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3b8f19-f99e-4299-a4bf-bd9cbabafc49" containerName="payload-processing" Oct 30 06:21:20.329694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.329663 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.353377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.353279 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4"] Oct 30 06:21:20.359475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.359450 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-dfa2ae-openshift-default-dockercfg-cqspr\"" Oct 30 06:21:20.496217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.496179 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c83045-0841-4f12-a77c-ada2ae280906-istiod-ca-cert\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.496656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.496632 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-credential-socket\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.496814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.496795 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-data\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.496975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.496957 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snnn2\" (UniqueName: \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-kube-api-access-snnn2\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.497158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.497142 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-envoy\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.497277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.497264 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-socket\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.497595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.497569 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-istio-token\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.497810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.497788 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-certs\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.497965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.497950 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c83045-0841-4f12-a77c-ada2ae280906-istio-podinfo\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.601646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.601549 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-envoy\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.601931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.601907 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-socket\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.602261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.602171 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-istio-token\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.602453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.602433 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-certs\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.602592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.602576 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c83045-0841-4f12-a77c-ada2ae280906-istio-podinfo\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.602759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.602743 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c83045-0841-4f12-a77c-ada2ae280906-istiod-ca-cert\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.602912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.602894 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-credential-socket\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.603055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.603039 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-data\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.603198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.603182 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snnn2\" (UniqueName: \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-kube-api-access-snnn2\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.641687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.641648 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-envoy\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.642519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.642496 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c83045-0841-4f12-a77c-ada2ae280906-istiod-ca-cert\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.643207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.643186 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-credential-socket\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.643601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.643582 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-data\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.653508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.653032 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-socket\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.653508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.653069 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snnn2\" (UniqueName: \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-kube-api-access-snnn2\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.653508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.653277 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-certs\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.683715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.683604 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-istio-token\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.683919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.683853 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c83045-0841-4f12-a77c-ada2ae280906-istio-podinfo\") pod \"e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:20.723750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.723714 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm"] Oct 30 06:21:20.724046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.723982 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" podUID="e9d46490-0091-4cb4-a80e-cd13b7ec2ea9" containerName="payload-processing" containerID="cri-o://023e9bced6c183cb8a3dddbddf14e9ff6608181b358edd38c5a7f58e5cd4d875" gracePeriod=30 Oct 30 06:21:20.785089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.784592 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt"] Oct 30 06:21:20.785089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.784911 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" podUID="64a0b9ed-f74f-4647-9956-664b0924db28" containerName="payload-pre-processing" containerID="cri-o://223b961376126b550517270b5241b7b9b447d855ab80b3c9eeb760610dbda399" gracePeriod=30 Oct 30 06:21:20.948351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:20.948183 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:21.233455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.229279 5001 generic.go:356] "Generic (PLEG): container finished" podID="e9d46490-0091-4cb4-a80e-cd13b7ec2ea9" containerID="023e9bced6c183cb8a3dddbddf14e9ff6608181b358edd38c5a7f58e5cd4d875" exitCode=0 Oct 30 06:21:21.233455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.229746 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" event={"ID":"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9","Type":"ContainerDied","Data":"023e9bced6c183cb8a3dddbddf14e9ff6608181b358edd38c5a7f58e5cd4d875"} Oct 30 06:21:21.284813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.284696 5001 generic.go:356] "Generic (PLEG): container finished" podID="64a0b9ed-f74f-4647-9956-664b0924db28" containerID="223b961376126b550517270b5241b7b9b447d855ab80b3c9eeb760610dbda399" exitCode=0 Oct 30 06:21:21.284813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.284776 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" event={"ID":"64a0b9ed-f74f-4647-9956-664b0924db28","Type":"ContainerDied","Data":"223b961376126b550517270b5241b7b9b447d855ab80b3c9eeb760610dbda399"} Oct 30 06:21:21.334134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.334074 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:21:21.405406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.405373 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:21:21.428138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.428111 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h"] Oct 30 06:21:21.433794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.433768 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bc5d8f-tdv6h"] Oct 30 06:21:21.478899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.478867 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a0b9ed-f74f-4647-9956-664b0924db28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a0b9ed-f74f-4647-9956-664b0924db28-plugins-config-volume\") pod \"64a0b9ed-f74f-4647-9956-664b0924db28\" (UID: \"64a0b9ed-f74f-4647-9956-664b0924db28\") " Oct 30 06:21:21.479081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.478966 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-plugins-config-volume\") pod \"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9\" (UID: \"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9\") " Oct 30 06:21:21.479081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.479018 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-kube-api-access-9ks6p\" (UniqueName: \"kubernetes.io/projected/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-kube-api-access-9ks6p\") pod \"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9\" (UID: \"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9\") " Oct 30 06:21:21.479204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.479132 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a0b9ed-f74f-4647-9956-664b0924db28-kube-api-access-lsp86\" (UniqueName: \"kubernetes.io/projected/64a0b9ed-f74f-4647-9956-664b0924db28-kube-api-access-lsp86\") pod \"64a0b9ed-f74f-4647-9956-664b0924db28\" (UID: \"64a0b9ed-f74f-4647-9956-664b0924db28\") " Oct 30 06:21:21.479373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.479342 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a0b9ed-f74f-4647-9956-664b0924db28-plugins-config-volume" pod "64a0b9ed-f74f-4647-9956-664b0924db28" (UID: "64a0b9ed-f74f-4647-9956-664b0924db28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:21.479517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.479408 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-plugins-config-volume" pod "e9d46490-0091-4cb4-a80e-cd13b7ec2ea9" (UID: "e9d46490-0091-4cb4-a80e-cd13b7ec2ea9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:21.479750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.479728 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a0b9ed-f74f-4647-9956-664b0924db28-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:21.479878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.479754 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:21.481647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.481618 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-kube-api-access-9ks6p" pod "e9d46490-0091-4cb4-a80e-cd13b7ec2ea9" (UID: "e9d46490-0091-4cb4-a80e-cd13b7ec2ea9"). InnerVolumeSpecName "kube-api-access-9ks6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:21.481877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.481856 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a0b9ed-f74f-4647-9956-664b0924db28-kube-api-access-lsp86" pod "64a0b9ed-f74f-4647-9956-664b0924db28" (UID: "64a0b9ed-f74f-4647-9956-664b0924db28"). InnerVolumeSpecName "kube-api-access-lsp86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:21.586405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.585615 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ks6p\" (UniqueName: \"kubernetes.io/projected/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9-kube-api-access-9ks6p\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:21.586405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.585656 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsp86\" (UniqueName: \"kubernetes.io/projected/64a0b9ed-f74f-4647-9956-664b0924db28-kube-api-access-lsp86\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:21.605716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.599882 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4"] Oct 30 06:21:21.630559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.630510 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:21.630675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.630615 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:21.630675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:21.630655 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:22.119401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.119365 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705f3a7b-4d9d-454f-aced-b12297375238" path="/var/lib/kubelet/pods/705f3a7b-4d9d-454f-aced-b12297375238/volumes" Oct 30 06:21:22.297872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.297838 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" event={"ID":"92c83045-0841-4f12-a77c-ada2ae280906","Type":"ContainerStarted","Data":"5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460"} Oct 30 06:21:22.298098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.297893 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" event={"ID":"92c83045-0841-4f12-a77c-ada2ae280906","Type":"ContainerStarted","Data":"2439c30daa77722a92a03306140ebc2ab575dc2eff490814ec66361d53af7db3"} Oct 30 06:21:22.302685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.299819 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" event={"ID":"64a0b9ed-f74f-4647-9956-664b0924db28","Type":"ContainerDied","Data":"56e790caeef92b2657f1580961fa378fd04ecde0094f8241f2f6d69685e42007"} Oct 30 06:21:22.302685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.299827 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt" Oct 30 06:21:22.302685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.299937 5001 scope.go:122] "RemoveContainer" containerID="223b961376126b550517270b5241b7b9b447d855ab80b3c9eeb760610dbda399" Oct 30 06:21:22.324530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.308754 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" event={"ID":"e9d46490-0091-4cb4-a80e-cd13b7ec2ea9","Type":"ContainerDied","Data":"4aa5a198ed655e21d4471d559dde75b341996a2dae80b6225679c4f1b8fb2c42"} Oct 30 06:21:22.324530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.308864 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm" Oct 30 06:21:22.349821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.349795 5001 scope.go:122] "RemoveContainer" containerID="023e9bced6c183cb8a3dddbddf14e9ff6608181b358edd38c5a7f58e5cd4d875" Oct 30 06:21:22.357740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.357683 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" podStartSLOduration=2.357663908 podStartE2EDuration="2.357663908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:22.351755996 +0000 UTC m=+1696.347713007" watchObservedRunningTime="2026-10-30 06:21:22.357663908 +0000 UTC m=+1696.353620923" Oct 30 06:21:22.390755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.390681 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm"] Oct 30 06:21:22.397247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.397221 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bc5d8f-755ff454bd-zz2pm"] Oct 30 06:21:22.428523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.428467 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt"] Oct 30 06:21:22.432333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.432292 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bc5d8f-6c6459c4c-s8wxt"] Oct 30 06:21:22.948489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.948458 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:22.953932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:22.953905 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:23.343256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.342145 5001 generic.go:356] "Generic (PLEG): container finished" podID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerID="fb9cc9241688c255e9cbb8ac6c0778f463d76e76adbbc82977642a2b6d76f651" exitCode=0 Oct 30 06:21:23.343256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.342218 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" event={"ID":"82997d1a-74de-4f2f-b8cb-47aa1fab159c","Type":"ContainerDied","Data":"fb9cc9241688c255e9cbb8ac6c0778f463d76e76adbbc82977642a2b6d76f651"} Oct 30 06:21:23.343256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.342280 5001 scope.go:122] "RemoveContainer" containerID="4a2e64bafe5b33136118b72d8e7eff8112424a287d34e962d7dc85a13908b56b" Oct 30 06:21:23.347348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.346540 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:23.347481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.347418 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:21:23.461909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.461875 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:21:23.531437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.531400 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-token\" (UniqueName: \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-token\") pod \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " Oct 30 06:21:23.532416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.532384 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-envoy\") pod \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " Oct 30 06:21:23.532571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.532493 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-podinfo\") pod \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " Oct 30 06:21:23.532675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.532656 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istiod-ca-cert\") pod \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " Oct 30 06:21:23.532738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.532721 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-credential-socket\") pod \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " Oct 30 06:21:23.532819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.532801 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-certs\") pod \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " Oct 30 06:21:23.532949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.532878 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-kube-api-access-v4hxb\" (UniqueName: \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-kube-api-access-v4hxb\") pod \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " Oct 30 06:21:23.533146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.533117 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-credential-socket" pod "82997d1a-74de-4f2f-b8cb-47aa1fab159c" (UID: "82997d1a-74de-4f2f-b8cb-47aa1fab159c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:23.533214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.533184 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-certs" pod "82997d1a-74de-4f2f-b8cb-47aa1fab159c" (UID: "82997d1a-74de-4f2f-b8cb-47aa1fab159c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:23.533274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.533217 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-data\") pod \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " Oct 30 06:21:23.533359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.533327 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-socket\") pod \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\" (UID: \"82997d1a-74de-4f2f-b8cb-47aa1fab159c\") " Oct 30 06:21:23.533582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.533557 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-data" pod "82997d1a-74de-4f2f-b8cb-47aa1fab159c" (UID: "82997d1a-74de-4f2f-b8cb-47aa1fab159c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:23.533660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.533641 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istiod-ca-cert" pod "82997d1a-74de-4f2f-b8cb-47aa1fab159c" (UID: "82997d1a-74de-4f2f-b8cb-47aa1fab159c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:23.533783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.533761 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-socket" pod "82997d1a-74de-4f2f-b8cb-47aa1fab159c" (UID: "82997d1a-74de-4f2f-b8cb-47aa1fab159c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:23.534031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.534007 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:23.534031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.534034 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:23.534216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.534046 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:23.534216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.534058 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:23.534216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.534071 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:23.540157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.540131 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-envoy" pod "82997d1a-74de-4f2f-b8cb-47aa1fab159c" (UID: "82997d1a-74de-4f2f-b8cb-47aa1fab159c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:23.540318 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.540187 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-token" pod "82997d1a-74de-4f2f-b8cb-47aa1fab159c" (UID: "82997d1a-74de-4f2f-b8cb-47aa1fab159c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:23.540407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.540321 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-podinfo" pod "82997d1a-74de-4f2f-b8cb-47aa1fab159c" (UID: "82997d1a-74de-4f2f-b8cb-47aa1fab159c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:23.541667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.541640 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-kube-api-access-v4hxb" pod "82997d1a-74de-4f2f-b8cb-47aa1fab159c" (UID: "82997d1a-74de-4f2f-b8cb-47aa1fab159c"). InnerVolumeSpecName "kube-api-access-v4hxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:23.640029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.636362 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4hxb\" (UniqueName: \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-kube-api-access-v4hxb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:23.640029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.636405 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:23.640029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.636419 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:23.640029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:23.636433 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82997d1a-74de-4f2f-b8cb-47aa1fab159c-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:24.126877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:24.126843 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a0b9ed-f74f-4647-9956-664b0924db28" path="/var/lib/kubelet/pods/64a0b9ed-f74f-4647-9956-664b0924db28/volumes" Oct 30 06:21:24.127271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:24.127257 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d46490-0091-4cb4-a80e-cd13b7ec2ea9" path="/var/lib/kubelet/pods/e9d46490-0091-4cb4-a80e-cd13b7ec2ea9/volumes" Oct 30 06:21:24.355651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:24.355614 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" Oct 30 06:21:24.355851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:24.355617 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b" event={"ID":"82997d1a-74de-4f2f-b8cb-47aa1fab159c","Type":"ContainerDied","Data":"88c6f3d79f2dff3c48caadc895a1b9013678e37a1b670521ccd0573c895d2889"} Oct 30 06:21:24.355851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:24.355798 5001 scope.go:122] "RemoveContainer" containerID="fb9cc9241688c255e9cbb8ac6c0778f463d76e76adbbc82977642a2b6d76f651" Oct 30 06:21:24.422408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:24.422321 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b"] Oct 30 06:21:24.434540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:24.434504 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a55a4b-openshift-default-687bcbd8d9-r4g9b"] Oct 30 06:21:26.118267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.118235 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" path="/var/lib/kubelet/pods/82997d1a-74de-4f2f-b8cb-47aa1fab159c/volumes" Oct 30 06:21:26.817985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.817954 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp"] Oct 30 06:21:26.818598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.818583 5001 cpu_manager.go:414] "containerMap: removing container" podUID="64a0b9ed-f74f-4647-9956-664b0924db28" containerName="payload-pre-processing" Oct 30 06:21:26.818677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.818599 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a0b9ed-f74f-4647-9956-664b0924db28" containerName="payload-pre-processing" Oct 30 06:21:26.818677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.818613 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e9d46490-0091-4cb4-a80e-cd13b7ec2ea9" containerName="payload-processing" Oct 30 06:21:26.818677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.818618 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d46490-0091-4cb4-a80e-cd13b7ec2ea9" containerName="payload-processing" Oct 30 06:21:26.818677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.818637 5001 cpu_manager.go:414] "containerMap: removing container" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" Oct 30 06:21:26.818677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.818641 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" Oct 30 06:21:26.818844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.818753 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" Oct 30 06:21:26.818844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.818763 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a0b9ed-f74f-4647-9956-664b0924db28" containerName="payload-pre-processing" Oct 30 06:21:26.818844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.818768 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d46490-0091-4cb4-a80e-cd13b7ec2ea9" containerName="payload-processing" Oct 30 06:21:26.822255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.822238 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:21:26.838524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.838498 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-dfa2ae\"" Oct 30 06:21:26.850081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.850052 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-dfa2ae-dockercfg-67jrw\"" Oct 30 06:21:26.855868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:26.855825 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp"] Oct 30 06:21:27.020338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.020282 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4c812d-c16f-4027-8413-f364014c1f1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp\" (UID: \"dc4c812d-c16f-4027-8413-f364014c1f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:21:27.020776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.020753 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2c5\" (UniqueName: \"kubernetes.io/projected/dc4c812d-c16f-4027-8413-f364014c1f1b-kube-api-access-zm2c5\") pod \"payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp\" (UID: \"dc4c812d-c16f-4027-8413-f364014c1f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:21:27.033507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.027174 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz"] Oct 30 06:21:27.033507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.028196 5001 cpu_manager.go:414] "containerMap: removing container" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" Oct 30 06:21:27.033507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.028215 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" Oct 30 06:21:27.033507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.028377 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="82997d1a-74de-4f2f-b8cb-47aa1fab159c" containerName="istio-proxy" Oct 30 06:21:27.033507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.033069 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:21:27.037944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.037747 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz"] Oct 30 06:21:27.122768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.122734 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2c5\" (UniqueName: \"kubernetes.io/projected/dc4c812d-c16f-4027-8413-f364014c1f1b-kube-api-access-zm2c5\") pod \"payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp\" (UID: \"dc4c812d-c16f-4027-8413-f364014c1f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:21:27.123259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.122942 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz\" (UID: \"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:21:27.123259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.122993 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4c812d-c16f-4027-8413-f364014c1f1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp\" (UID: \"dc4c812d-c16f-4027-8413-f364014c1f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:21:27.123259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.123045 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvct\" (UniqueName: \"kubernetes.io/projected/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-kube-api-access-hpvct\") pod \"payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz\" (UID: \"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:21:27.124362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.124337 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4c812d-c16f-4027-8413-f364014c1f1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp\" (UID: \"dc4c812d-c16f-4027-8413-f364014c1f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:21:27.159355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.158683 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2c5\" (UniqueName: \"kubernetes.io/projected/dc4c812d-c16f-4027-8413-f364014c1f1b-kube-api-access-zm2c5\") pod \"payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp\" (UID: \"dc4c812d-c16f-4027-8413-f364014c1f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:21:27.224400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.224366 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz\" (UID: \"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:21:27.224617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.224452 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvct\" (UniqueName: \"kubernetes.io/projected/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-kube-api-access-hpvct\") pod \"payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz\" (UID: \"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:21:27.225089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.225070 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz\" (UID: \"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:21:27.259895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.259864 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvct\" (UniqueName: \"kubernetes.io/projected/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-kube-api-access-hpvct\") pod \"payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz\" (UID: \"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:21:27.415403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.415371 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:21:27.436981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.436950 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:21:27.769619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.769587 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz"] Oct 30 06:21:27.770597 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:21:27.770566 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf6d2cb_7ed8_4375_b0da_73a49945b0fb.slice/crio-dd77f57fc67ef51d8320524f886215e441ddeba4a0cfc0b362098866939f31f5 WatchSource:0}: Error finding container dd77f57fc67ef51d8320524f886215e441ddeba4a0cfc0b362098866939f31f5: Status 404 returned error can't find the container with id dd77f57fc67ef51d8320524f886215e441ddeba4a0cfc0b362098866939f31f5 Oct 30 06:21:27.791091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:27.791062 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp"] Oct 30 06:21:27.791648 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:21:27.791582 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4c812d_c16f_4027_8413_f364014c1f1b.slice/crio-4890dcab0438dbca20240c50c5e306209523f32bb0a68059b590fc357654bebe WatchSource:0}: Error finding container 4890dcab0438dbca20240c50c5e306209523f32bb0a68059b590fc357654bebe: Status 404 returned error can't find the container with id 4890dcab0438dbca20240c50c5e306209523f32bb0a68059b590fc357654bebe Oct 30 06:21:28.391319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:28.391265 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" event={"ID":"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb","Type":"ContainerStarted","Data":"29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba"} Oct 30 06:21:28.391836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:28.391331 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" event={"ID":"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb","Type":"ContainerStarted","Data":"dd77f57fc67ef51d8320524f886215e441ddeba4a0cfc0b362098866939f31f5"} Oct 30 06:21:28.391836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:28.391363 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:21:28.393476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:28.393428 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" event={"ID":"dc4c812d-c16f-4027-8413-f364014c1f1b","Type":"ContainerStarted","Data":"4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab"} Oct 30 06:21:28.393611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:28.393481 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" event={"ID":"dc4c812d-c16f-4027-8413-f364014c1f1b","Type":"ContainerStarted","Data":"4890dcab0438dbca20240c50c5e306209523f32bb0a68059b590fc357654bebe"} Oct 30 06:21:28.393611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:28.393507 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:21:28.435210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:28.435162 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" podStartSLOduration=2.435148835 podStartE2EDuration="2.435148835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:28.431718337 +0000 UTC m=+1702.427675348" watchObservedRunningTime="2026-10-30 06:21:28.435148835 +0000 UTC m=+1702.431105875" Oct 30 06:21:28.474162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:28.474107 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" podStartSLOduration=2.474093573 podStartE2EDuration="2.474093573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:28.470467631 +0000 UTC m=+1702.466424642" watchObservedRunningTime="2026-10-30 06:21:28.474093573 +0000 UTC m=+1702.470050574" Oct 30 06:21:30.009612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:30.009573 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l"] Oct 30 06:21:30.010111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:30.009841 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" containerID="cri-o://c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006" gracePeriod=30 Oct 30 06:21:32.241969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.241931 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg"] Oct 30 06:21:32.247458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.247433 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.287332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.273678 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-93c22e-openshift-default-dockercfg-28xg6\"" Oct 30 06:21:32.298064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.298027 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-data\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.298237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.298079 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vcbf\" (UniqueName: \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-kube-api-access-2vcbf\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.298237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.298118 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-socket\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.298237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.298230 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-credential-socket\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.298448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.298333 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-certs\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.298448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.298362 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-podinfo\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.298448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.298413 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-envoy\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.298615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.298482 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istiod-ca-cert\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.298615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.298505 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-token\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.307317 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.307258 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg"] Oct 30 06:21:32.399980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.399947 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-socket\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.400174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.400035 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-credential-socket\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.400174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.400091 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-certs\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.400174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.400114 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-podinfo\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.400174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.400159 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-envoy\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.400454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.400219 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istiod-ca-cert\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.400454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.400241 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-token\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.400454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.400348 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-data\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.400454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.400375 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vcbf\" (UniqueName: \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-kube-api-access-2vcbf\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.400664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.400481 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-socket\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.400664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.400502 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-credential-socket\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.400880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.400852 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-data\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.401116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.401092 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-certs\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.401418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.401399 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istiod-ca-cert\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.403285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.403257 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-envoy\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.424470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.409879 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-podinfo\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.457344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.457287 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-token\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.481206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.480062 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vcbf\" (UniqueName: \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-kube-api-access-2vcbf\") pod \"e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:32.618623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:32.618587 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:33.005896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:33.005869 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg"] Oct 30 06:21:33.007728 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:21:33.007702 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ac07ad_23ba_47a7_bf7f_4bcd9315ff75.slice/crio-ca8e1d26ecebb696c10505982f6a648ada3d553cf8d25fd583010b14b2ae6d89 WatchSource:0}: Error finding container ca8e1d26ecebb696c10505982f6a648ada3d553cf8d25fd583010b14b2ae6d89: Status 404 returned error can't find the container with id ca8e1d26ecebb696c10505982f6a648ada3d553cf8d25fd583010b14b2ae6d89 Oct 30 06:21:33.010272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:33.010236 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:33.010392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:33.010326 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:33.010392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:33.010358 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:33.437386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:33.437352 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:21:33.437797 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:33.437429 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:21:33.584975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:33.584922 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" event={"ID":"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75","Type":"ContainerStarted","Data":"e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89"} Oct 30 06:21:33.585168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:33.584990 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" event={"ID":"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75","Type":"ContainerStarted","Data":"ca8e1d26ecebb696c10505982f6a648ada3d553cf8d25fd583010b14b2ae6d89"} Oct 30 06:21:33.619256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:33.619225 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:33.643179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:33.643131 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" podStartSLOduration=1.643117003 podStartE2EDuration="1.643117003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:33.640738689 +0000 UTC m=+1707.636695699" watchObservedRunningTime="2026-10-30 06:21:33.643117003 +0000 UTC m=+1707.639074074" Oct 30 06:21:34.675332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:34.675071 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:35.389799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.389771 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:21:35.446460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.446428 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63052f6-d65d-4307-93bb-daa7a5581e78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63052f6-d65d-4307-93bb-daa7a5581e78-istiod-ca-cert\") pod \"a63052f6-d65d-4307-93bb-daa7a5581e78\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " Oct 30 06:21:35.446736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.446476 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-credential-socket\") pod \"a63052f6-d65d-4307-93bb-daa7a5581e78\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " Oct 30 06:21:35.446736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.446515 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-envoy\") pod \"a63052f6-d65d-4307-93bb-daa7a5581e78\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " Oct 30 06:21:35.446869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.446755 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-token\" (UniqueName: \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-token\") pod \"a63052f6-d65d-4307-93bb-daa7a5581e78\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " Oct 30 06:21:35.446869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.446788 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-kube-api-access-kqv8z\" (UniqueName: \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-kube-api-access-kqv8z\") pod \"a63052f6-d65d-4307-93bb-daa7a5581e78\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " Oct 30 06:21:35.446869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.446780 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-credential-socket" pod "a63052f6-d65d-4307-93bb-daa7a5581e78" (UID: "a63052f6-d65d-4307-93bb-daa7a5581e78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:35.446973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.446919 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63052f6-d65d-4307-93bb-daa7a5581e78-istiod-ca-cert" pod "a63052f6-d65d-4307-93bb-daa7a5581e78" (UID: "a63052f6-d65d-4307-93bb-daa7a5581e78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:35.446973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.446904 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-podinfo\") pod \"a63052f6-d65d-4307-93bb-daa7a5581e78\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " Oct 30 06:21:35.447043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.446970 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-certs\") pod \"a63052f6-d65d-4307-93bb-daa7a5581e78\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " Oct 30 06:21:35.447511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.447088 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-socket\") pod \"a63052f6-d65d-4307-93bb-daa7a5581e78\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " Oct 30 06:21:35.447511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.447246 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-data\") pod \"a63052f6-d65d-4307-93bb-daa7a5581e78\" (UID: \"a63052f6-d65d-4307-93bb-daa7a5581e78\") " Oct 30 06:21:35.447511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.447277 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-certs" pod "a63052f6-d65d-4307-93bb-daa7a5581e78" (UID: "a63052f6-d65d-4307-93bb-daa7a5581e78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:35.448934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.447506 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-socket" pod "a63052f6-d65d-4307-93bb-daa7a5581e78" (UID: "a63052f6-d65d-4307-93bb-daa7a5581e78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:35.448934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.447939 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-data" pod "a63052f6-d65d-4307-93bb-daa7a5581e78" (UID: "a63052f6-d65d-4307-93bb-daa7a5581e78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:35.449660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.449566 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:35.449660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.449592 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:35.449660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.449607 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:35.449660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.449620 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63052f6-d65d-4307-93bb-daa7a5581e78-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:35.449660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.449631 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:35.450034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.450007 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-envoy" pod "a63052f6-d65d-4307-93bb-daa7a5581e78" (UID: "a63052f6-d65d-4307-93bb-daa7a5581e78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:35.450177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.450139 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-podinfo" pod "a63052f6-d65d-4307-93bb-daa7a5581e78" (UID: "a63052f6-d65d-4307-93bb-daa7a5581e78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:35.450290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.450229 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-token" pod "a63052f6-d65d-4307-93bb-daa7a5581e78" (UID: "a63052f6-d65d-4307-93bb-daa7a5581e78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:35.450618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.450593 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-kube-api-access-kqv8z" pod "a63052f6-d65d-4307-93bb-daa7a5581e78" (UID: "a63052f6-d65d-4307-93bb-daa7a5581e78"). InnerVolumeSpecName "kube-api-access-kqv8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:35.550804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.550774 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:35.550804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.550801 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:35.550804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.550810 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqv8z\" (UniqueName: \"kubernetes.io/projected/a63052f6-d65d-4307-93bb-daa7a5581e78-kube-api-access-kqv8z\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:35.551046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.550820 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63052f6-d65d-4307-93bb-daa7a5581e78-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:21:35.605888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.605855 5001 generic.go:356] "Generic (PLEG): container finished" podID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerID="c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006" exitCode=0 Oct 30 06:21:35.606077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.605917 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" Oct 30 06:21:35.606077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.605944 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" event={"ID":"a63052f6-d65d-4307-93bb-daa7a5581e78","Type":"ContainerDied","Data":"c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006"} Oct 30 06:21:35.606077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.606001 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l" event={"ID":"a63052f6-d65d-4307-93bb-daa7a5581e78","Type":"ContainerDied","Data":"a77462158d1728d4a8424134765916b784bfa410f4f2223a30a2e6405ee6e956"} Oct 30 06:21:35.606077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.606030 5001 scope.go:122] "RemoveContainer" containerID="c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006" Oct 30 06:21:35.606660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.606635 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:35.613343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.608490 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:21:35.628971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.628947 5001 scope.go:122] "RemoveContainer" containerID="82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d" Oct 30 06:21:35.639506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.639485 5001 scope.go:122] "RemoveContainer" containerID="c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006" Oct 30 06:21:35.639783 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:21:35.639764 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006\": container with ID starting with c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006 not found: ID does not exist" containerID="c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006" Oct 30 06:21:35.639827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.639791 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006"} err="failed to get container status \"c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006\": rpc error: code = NotFound desc = could not find container \"c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006\": container with ID starting with c2fa0e90d1f6b15a5592e7c7e456d4b6f57cc1adbe7c5a25024963a851613006 not found: ID does not exist" Oct 30 06:21:35.639827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.639806 5001 scope.go:122] "RemoveContainer" containerID="82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d" Oct 30 06:21:35.640028 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:21:35.640013 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d\": container with ID starting with 82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d not found: ID does not exist" containerID="82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d" Oct 30 06:21:35.640070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.640031 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d"} err="failed to get container status \"82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d\": rpc error: code = NotFound desc = could not find container \"82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d\": container with ID starting with 82795c33754b4dfea85146f36546a11149852c79da3db84d40ae29fb9e45fc1d not found: ID does not exist" Oct 30 06:21:35.717333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.716675 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l"] Oct 30 06:21:35.723374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:35.720193 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-bc5d8f-openshift-default-65566bcdd5-2wk4l"] Oct 30 06:21:36.119215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.119134 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" path="/var/lib/kubelet/pods/a63052f6-d65d-4307-93bb-daa7a5581e78/volumes" Oct 30 06:21:36.469885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.469801 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk"] Oct 30 06:21:36.470791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.470769 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" Oct 30 06:21:36.470912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.470793 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" Oct 30 06:21:36.470972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.470961 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" Oct 30 06:21:36.471031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.470976 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" Oct 30 06:21:36.500557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.500523 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.517007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.516982 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk"] Oct 30 06:21:36.522544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.522512 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-4dd310-openshift-default-dockercfg-rj5zz\"" Oct 30 06:21:36.566793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.566757 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istiod-ca-cert\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.566985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.566834 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-socket\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.566985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.566889 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-podinfo\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.566985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.566915 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-envoy\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.567174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.567008 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-token\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.567174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.567075 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-data\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.567174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.567154 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-credential-socket\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.567345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.567216 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mnqg\" (UniqueName: \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-kube-api-access-4mnqg\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.567345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.567251 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-certs\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.674757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.674112 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istiod-ca-cert\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.674757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.674192 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-socket\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.674757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.674239 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-podinfo\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.674757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.674264 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-envoy\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.674757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.674377 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-token\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.674757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.674442 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-data\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.674757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.674520 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-credential-socket\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.674757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.674580 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mnqg\" (UniqueName: \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-kube-api-access-4mnqg\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.674757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.674612 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-certs\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.675535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.674770 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-socket\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.675535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.674996 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-certs\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.675535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.675440 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istiod-ca-cert\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.675731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.675681 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-data\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.692329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.686027 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-podinfo\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.696160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.695711 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-envoy\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.696160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.695861 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-credential-socket\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.745376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.741205 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mnqg\" (UniqueName: \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-kube-api-access-4mnqg\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.746645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.746529 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-token\") pod \"e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:36.819963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:36.819602 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:37.197131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:37.197098 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk"] Oct 30 06:21:37.198136 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:21:37.198092 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb19e5cd_434e_4d2d_bf69_ab0412742d3c.slice/crio-dd02b27b3c27017d01c5db7c2c922ea8ae1d8d679817e2ef4ce0c8289ccc3d9b WatchSource:0}: Error finding container dd02b27b3c27017d01c5db7c2c922ea8ae1d8d679817e2ef4ce0c8289ccc3d9b: Status 404 returned error can't find the container with id dd02b27b3c27017d01c5db7c2c922ea8ae1d8d679817e2ef4ce0c8289ccc3d9b Oct 30 06:21:37.201499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:37.201459 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:37.201586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:37.201543 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:37.201632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:37.201584 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:37.706473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:37.706281 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" event={"ID":"eb19e5cd-434e-4d2d-bf69-ab0412742d3c","Type":"ContainerStarted","Data":"8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3"} Oct 30 06:21:37.706473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:37.706384 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" event={"ID":"eb19e5cd-434e-4d2d-bf69-ab0412742d3c","Type":"ContainerStarted","Data":"dd02b27b3c27017d01c5db7c2c922ea8ae1d8d679817e2ef4ce0c8289ccc3d9b"} Oct 30 06:21:37.775756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:37.775699 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" podStartSLOduration=1.775682749 podStartE2EDuration="1.775682749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:37.761912821 +0000 UTC m=+1711.757869832" watchObservedRunningTime="2026-10-30 06:21:37.775682749 +0000 UTC m=+1711.771639759" Oct 30 06:21:37.821299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:37.821261 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:38.129188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.129158 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r"] Oct 30 06:21:38.129945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.129923 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" Oct 30 06:21:38.130079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.129945 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63052f6-d65d-4307-93bb-daa7a5581e78" containerName="istio-proxy" Oct 30 06:21:38.133995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.133969 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:21:38.148746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.148716 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-93c22e\"" Oct 30 06:21:38.151390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.151364 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r"] Oct 30 06:21:38.151686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.151652 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-93c22e-dockercfg-stfp4\"" Oct 30 06:21:38.404133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.398707 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnv9m\" (UniqueName: \"kubernetes.io/projected/e6486dc4-552e-4e15-a6f4-24228916eeaf-kube-api-access-jnv9m\") pod \"payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r\" (UID: \"e6486dc4-552e-4e15-a6f4-24228916eeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:21:38.404133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.398917 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6486dc4-552e-4e15-a6f4-24228916eeaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r\" (UID: \"e6486dc4-552e-4e15-a6f4-24228916eeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:21:38.449891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.449856 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw"] Oct 30 06:21:38.460570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.460538 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:21:38.485162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.485111 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw"] Oct 30 06:21:38.513366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.500918 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6486dc4-552e-4e15-a6f4-24228916eeaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r\" (UID: \"e6486dc4-552e-4e15-a6f4-24228916eeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:21:38.513366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.501073 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnv9m\" (UniqueName: \"kubernetes.io/projected/e6486dc4-552e-4e15-a6f4-24228916eeaf-kube-api-access-jnv9m\") pod \"payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r\" (UID: \"e6486dc4-552e-4e15-a6f4-24228916eeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:21:38.513366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.502192 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6486dc4-552e-4e15-a6f4-24228916eeaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r\" (UID: \"e6486dc4-552e-4e15-a6f4-24228916eeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:21:38.555881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.555802 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnv9m\" (UniqueName: \"kubernetes.io/projected/e6486dc4-552e-4e15-a6f4-24228916eeaf-kube-api-access-jnv9m\") pod \"payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r\" (UID: \"e6486dc4-552e-4e15-a6f4-24228916eeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:21:38.602776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.602220 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkqj\" (UniqueName: \"kubernetes.io/projected/42deeec4-aaad-44a6-bdf9-9a46893a16d7-kube-api-access-8mkqj\") pod \"payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw\" (UID: \"42deeec4-aaad-44a6-bdf9-9a46893a16d7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:21:38.602776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.602502 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42deeec4-aaad-44a6-bdf9-9a46893a16d7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw\" (UID: \"42deeec4-aaad-44a6-bdf9-9a46893a16d7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:21:38.626893 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:21:38.626853 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:21:38.707701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.706537 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42deeec4-aaad-44a6-bdf9-9a46893a16d7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw\" (UID: \"42deeec4-aaad-44a6-bdf9-9a46893a16d7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:21:38.707701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.706710 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkqj\" (UniqueName: \"kubernetes.io/projected/42deeec4-aaad-44a6-bdf9-9a46893a16d7-kube-api-access-8mkqj\") pod \"payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw\" (UID: \"42deeec4-aaad-44a6-bdf9-9a46893a16d7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:21:38.707701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.707587 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42deeec4-aaad-44a6-bdf9-9a46893a16d7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw\" (UID: \"42deeec4-aaad-44a6-bdf9-9a46893a16d7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:21:38.730941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.730905 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkqj\" (UniqueName: \"kubernetes.io/projected/42deeec4-aaad-44a6-bdf9-9a46893a16d7-kube-api-access-8mkqj\") pod \"payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw\" (UID: \"42deeec4-aaad-44a6-bdf9-9a46893a16d7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:21:38.750255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.750218 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:21:38.813008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.812976 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:21:38.845790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:38.845523 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:39.071578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.070834 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r"] Oct 30 06:21:39.075595 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:21:39.075563 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6486dc4_552e_4e15_a6f4_24228916eeaf.slice/crio-097a3d74a94dffbd29b688080b3fc15ec98aa933e32e8e4470b5ac60f3d6a794 WatchSource:0}: Error finding container 097a3d74a94dffbd29b688080b3fc15ec98aa933e32e8e4470b5ac60f3d6a794: Status 404 returned error can't find the container with id 097a3d74a94dffbd29b688080b3fc15ec98aa933e32e8e4470b5ac60f3d6a794 Oct 30 06:21:39.157353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.157296 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw"] Oct 30 06:21:39.158946 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:21:39.158921 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42deeec4_aaad_44a6_bdf9_9a46893a16d7.slice/crio-b2c6a5ce493274e7f48a923a32f0fe1debfe87a14f11f047f6faa810700aa52b WatchSource:0}: Error finding container b2c6a5ce493274e7f48a923a32f0fe1debfe87a14f11f047f6faa810700aa52b: Status 404 returned error can't find the container with id b2c6a5ce493274e7f48a923a32f0fe1debfe87a14f11f047f6faa810700aa52b Oct 30 06:21:39.468800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.468763 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:21:39.468981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.468848 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:21:39.731824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.731731 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" event={"ID":"42deeec4-aaad-44a6-bdf9-9a46893a16d7","Type":"ContainerStarted","Data":"4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7"} Oct 30 06:21:39.731824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.731798 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:21:39.731824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.731822 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" event={"ID":"42deeec4-aaad-44a6-bdf9-9a46893a16d7","Type":"ContainerStarted","Data":"b2c6a5ce493274e7f48a923a32f0fe1debfe87a14f11f047f6faa810700aa52b"} Oct 30 06:21:39.733719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.733645 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" event={"ID":"e6486dc4-552e-4e15-a6f4-24228916eeaf","Type":"ContainerStarted","Data":"2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442"} Oct 30 06:21:39.734242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.734221 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:21:39.734372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.734266 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:39.734372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.734294 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" event={"ID":"e6486dc4-552e-4e15-a6f4-24228916eeaf","Type":"ContainerStarted","Data":"097a3d74a94dffbd29b688080b3fc15ec98aa933e32e8e4470b5ac60f3d6a794"} Oct 30 06:21:39.735168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.735148 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:21:39.776425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.776362 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" podStartSLOduration=1.7763433 podStartE2EDuration="1.7763433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:39.774317016 +0000 UTC m=+1713.770274022" watchObservedRunningTime="2026-10-30 06:21:39.7763433 +0000 UTC m=+1713.772300313" Oct 30 06:21:39.863679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:39.863620 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" podStartSLOduration=1.8635995969999999 podStartE2EDuration="1.863599597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:39.861747655 +0000 UTC m=+1713.857704665" watchObservedRunningTime="2026-10-30 06:21:39.863599597 +0000 UTC m=+1713.859556608" Oct 30 06:21:41.424846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:41.424806 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:21:41.425280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:41.424862 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:21:41.755283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:41.755197 5001 generic.go:356] "Generic (PLEG): container finished" podID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerID="bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b" exitCode=0 Oct 30 06:21:41.755283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:41.755275 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" event={"ID":"975f7621-69e1-4a2c-8f93-871c39b4ca75","Type":"ContainerDied","Data":"bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b"} Oct 30 06:21:41.755836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:41.755808 5001 scope.go:122] "RemoveContainer" containerID="bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b" Oct 30 06:21:41.757486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:41.757458 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:41.757561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:41.757528 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:41.757605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:41.757563 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:42.173683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.173647 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr"] Oct 30 06:21:42.178902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.178877 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:21:42.194266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.194225 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4dd310\"" Oct 30 06:21:42.198516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.198492 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4dd310-dockercfg-jqlft\"" Oct 30 06:21:42.199723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.199698 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr"] Oct 30 06:21:42.235898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.235852 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb"] Oct 30 06:21:42.241399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.241369 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:21:42.267908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.267873 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb"] Oct 30 06:21:42.322332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.300537 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec461549-4c1d-4a0e-ab11-baf044529086-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb\" (UID: \"ec461549-4c1d-4a0e-ab11-baf044529086\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:21:42.501506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.501412 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8556h\" (UniqueName: \"kubernetes.io/projected/ec461549-4c1d-4a0e-ab11-baf044529086-kube-api-access-8556h\") pod \"payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb\" (UID: \"ec461549-4c1d-4a0e-ab11-baf044529086\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:21:42.502432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.502239 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd08ddf-057c-49ac-a89a-0250c6c3867f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr\" (UID: \"bdd08ddf-057c-49ac-a89a-0250c6c3867f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:21:42.502659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.502634 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsvct\" (UniqueName: \"kubernetes.io/projected/bdd08ddf-057c-49ac-a89a-0250c6c3867f-kube-api-access-hsvct\") pod \"payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr\" (UID: \"bdd08ddf-057c-49ac-a89a-0250c6c3867f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:21:42.503070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.503043 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec461549-4c1d-4a0e-ab11-baf044529086-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb\" (UID: \"ec461549-4c1d-4a0e-ab11-baf044529086\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:21:42.504111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.504087 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec461549-4c1d-4a0e-ab11-baf044529086-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb\" (UID: \"ec461549-4c1d-4a0e-ab11-baf044529086\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:21:42.606191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.606150 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8556h\" (UniqueName: \"kubernetes.io/projected/ec461549-4c1d-4a0e-ab11-baf044529086-kube-api-access-8556h\") pod \"payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb\" (UID: \"ec461549-4c1d-4a0e-ab11-baf044529086\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:21:42.606419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.606210 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd08ddf-057c-49ac-a89a-0250c6c3867f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr\" (UID: \"bdd08ddf-057c-49ac-a89a-0250c6c3867f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:21:42.606419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.606246 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsvct\" (UniqueName: \"kubernetes.io/projected/bdd08ddf-057c-49ac-a89a-0250c6c3867f-kube-api-access-hsvct\") pod \"payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr\" (UID: \"bdd08ddf-057c-49ac-a89a-0250c6c3867f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:21:42.606994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.606974 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd08ddf-057c-49ac-a89a-0250c6c3867f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr\" (UID: \"bdd08ddf-057c-49ac-a89a-0250c6c3867f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:21:42.629940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.629906 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsvct\" (UniqueName: \"kubernetes.io/projected/bdd08ddf-057c-49ac-a89a-0250c6c3867f-kube-api-access-hsvct\") pod \"payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr\" (UID: \"bdd08ddf-057c-49ac-a89a-0250c6c3867f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:21:42.631940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.631907 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8556h\" (UniqueName: \"kubernetes.io/projected/ec461549-4c1d-4a0e-ab11-baf044529086-kube-api-access-8556h\") pod \"payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb\" (UID: \"ec461549-4c1d-4a0e-ab11-baf044529086\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:21:42.792877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.792785 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" event={"ID":"975f7621-69e1-4a2c-8f93-871c39b4ca75","Type":"ContainerStarted","Data":"185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356"} Oct 30 06:21:42.798193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.798168 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:21:42.873332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:42.865564 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:21:43.046482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.045511 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:21:43.046482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.045773 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:21:43.046482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.045823 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:21:43.188823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.188788 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr"] Oct 30 06:21:43.296172 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.294906 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb"] Oct 30 06:21:43.299017 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:21:43.298956 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec461549_4c1d_4a0e_ab11_baf044529086.slice/crio-2edd0a32f80b46595c69288dcff149b2436cc0aac2456c2a7e2c5c96e60cfac4 WatchSource:0}: Error finding container 2edd0a32f80b46595c69288dcff149b2436cc0aac2456c2a7e2c5c96e60cfac4: Status 404 returned error can't find the container with id 2edd0a32f80b46595c69288dcff149b2436cc0aac2456c2a7e2c5c96e60cfac4 Oct 30 06:21:43.802516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.802479 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" event={"ID":"ec461549-4c1d-4a0e-ab11-baf044529086","Type":"ContainerStarted","Data":"d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd"} Oct 30 06:21:43.803027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.802542 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" event={"ID":"ec461549-4c1d-4a0e-ab11-baf044529086","Type":"ContainerStarted","Data":"2edd0a32f80b46595c69288dcff149b2436cc0aac2456c2a7e2c5c96e60cfac4"} Oct 30 06:21:43.803027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.802580 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:21:43.804452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.804429 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" event={"ID":"bdd08ddf-057c-49ac-a89a-0250c6c3867f","Type":"ContainerStarted","Data":"c1129d7cba57576c309762d481bb8d23e4a19f2ebb0b0e2ece28d3de07335d8e"} Oct 30 06:21:43.804566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.804478 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" event={"ID":"bdd08ddf-057c-49ac-a89a-0250c6c3867f","Type":"ContainerStarted","Data":"2ea0d30ec3cd308a4f0359edc99da8f717a367dd20aa64a8c133f18bbc8e66f1"} Oct 30 06:21:43.804672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.804659 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:21:43.843162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.843108 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" podStartSLOduration=1.843094742 podStartE2EDuration="1.843094742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:43.841204733 +0000 UTC m=+1717.837161742" watchObservedRunningTime="2026-10-30 06:21:43.843094742 +0000 UTC m=+1717.839051751" Oct 30 06:21:43.891637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:43.891582 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" podStartSLOduration=1.891567785 podStartE2EDuration="1.891567785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:43.889858196 +0000 UTC m=+1717.885815205" watchObservedRunningTime="2026-10-30 06:21:43.891567785 +0000 UTC m=+1717.887524791" Oct 30 06:21:44.056417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:44.052726 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:21:44.056417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:44.052786 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:21:45.046789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:45.046748 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:21:45.047524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:45.047489 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:21:46.042555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:46.042518 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:21:46.042777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:46.042585 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:21:47.042892 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:47.042851 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:21:47.043513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:47.042910 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:21:48.042613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:48.042580 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:21:48.042805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:48.042645 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:21:49.047569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.047535 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:21:49.048022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.047949 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:21:49.048805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.048788 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:21:49.243404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.243370 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng"] Oct 30 06:21:49.254592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.254564 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.273652 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.273623 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f6a022-openshift-default-dockercfg-mbcbj\"" Oct 30 06:21:49.276182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.276158 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng"] Oct 30 06:21:49.291087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.291043 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448e441c-2924-4232-93c1-721a7b633880-istiod-ca-cert\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.291250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.291111 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-socket\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.291250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.291151 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-credential-socket\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.291250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.291190 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-istio-token\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.291250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.291230 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-data\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.291521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.291266 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-envoy\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.291521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.291371 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448e441c-2924-4232-93c1-721a7b633880-istio-podinfo\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.291521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.291453 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n6mx\" (UniqueName: \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-kube-api-access-7n6mx\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.291689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.291536 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-certs\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.392476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.392439 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n6mx\" (UniqueName: \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-kube-api-access-7n6mx\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.392685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.392545 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-certs\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.392685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.392617 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448e441c-2924-4232-93c1-721a7b633880-istiod-ca-cert\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.392685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.392672 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-socket\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.392869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.392712 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-credential-socket\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.392869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.392749 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-istio-token\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.392869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.392790 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-data\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.392869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.392826 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-envoy\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.393169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.392918 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448e441c-2924-4232-93c1-721a7b633880-istio-podinfo\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.393169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.393013 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-certs\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.393169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.393079 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-socket\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.393638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.393393 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-data\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.393756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.393702 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448e441c-2924-4232-93c1-721a7b633880-istiod-ca-cert\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.394254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.394151 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-credential-socket\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.396559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.396534 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448e441c-2924-4232-93c1-721a7b633880-istio-podinfo\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.401026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.400991 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-envoy\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.415491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.415469 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n6mx\" (UniqueName: \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-kube-api-access-7n6mx\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.415592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.415565 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-istio-token\") pod \"e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:49.576594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:49.576559 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:50.023133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:50.023100 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng"] Oct 30 06:21:50.024637 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:21:50.024614 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448e441c_2924_4232_93c1_721a7b633880.slice/crio-7962a7d43910a391c7cfecd84ca10eadd6d2ba1a833fc1926fa844b128c20bcf WatchSource:0}: Error finding container 7962a7d43910a391c7cfecd84ca10eadd6d2ba1a833fc1926fa844b128c20bcf: Status 404 returned error can't find the container with id 7962a7d43910a391c7cfecd84ca10eadd6d2ba1a833fc1926fa844b128c20bcf Oct 30 06:21:50.027051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:50.027020 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:50.027121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:50.027085 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:50.027121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:50.027115 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:21:50.748131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:50.748103 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:21:50.871844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:50.869677 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" event={"ID":"448e441c-2924-4232-93c1-721a7b633880","Type":"ContainerStarted","Data":"56caec93d7705a6453a81c29e86f5f34c576b1d66eb32d979b7373c9088b8209"} Oct 30 06:21:50.871844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:50.869739 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" event={"ID":"448e441c-2924-4232-93c1-721a7b633880","Type":"ContainerStarted","Data":"7962a7d43910a391c7cfecd84ca10eadd6d2ba1a833fc1926fa844b128c20bcf"} Oct 30 06:21:50.916831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:50.916777 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" podStartSLOduration=1.9167636030000001 podStartE2EDuration="1.916763603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:50.913973725 +0000 UTC m=+1724.909930736" watchObservedRunningTime="2026-10-30 06:21:50.916763603 +0000 UTC m=+1724.912720612" Oct 30 06:21:51.577559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:51.577514 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:51.583546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:51.583520 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:51.759575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:51.759547 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:21:51.876791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:51.876747 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:51.878076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:51.878052 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:21:54.116330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:54.110198 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:21:54.816915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:54.816887 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:21:54.817121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:54.816967 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:21:55.574336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:55.573083 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj"] Oct 30 06:21:55.579799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:55.579774 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:21:55.598897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:55.598867 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj"] Oct 30 06:21:55.610566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:55.610532 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f6a022-dockercfg-ls5vh\"" Oct 30 06:21:55.610566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:55.610532 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f6a022\"" Oct 30 06:21:55.716329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:55.714110 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8821eff-dc4e-456a-9112-5fee294c08ac-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj\" (UID: \"c8821eff-dc4e-456a-9112-5fee294c08ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:21:55.716329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:55.714379 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b95r2\" (UniqueName: \"kubernetes.io/projected/c8821eff-dc4e-456a-9112-5fee294c08ac-kube-api-access-b95r2\") pod \"payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj\" (UID: \"c8821eff-dc4e-456a-9112-5fee294c08ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:21:55.864102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:55.859670 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95r2\" (UniqueName: \"kubernetes.io/projected/c8821eff-dc4e-456a-9112-5fee294c08ac-kube-api-access-b95r2\") pod \"payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj\" (UID: \"c8821eff-dc4e-456a-9112-5fee294c08ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:21:55.864102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:55.859804 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8821eff-dc4e-456a-9112-5fee294c08ac-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj\" (UID: \"c8821eff-dc4e-456a-9112-5fee294c08ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:21:55.864102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:55.860932 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8821eff-dc4e-456a-9112-5fee294c08ac-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj\" (UID: \"c8821eff-dc4e-456a-9112-5fee294c08ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:21:55.945973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:55.945932 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95r2\" (UniqueName: \"kubernetes.io/projected/c8821eff-dc4e-456a-9112-5fee294c08ac-kube-api-access-b95r2\") pod \"payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj\" (UID: \"c8821eff-dc4e-456a-9112-5fee294c08ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:21:56.196230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:56.196139 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:21:56.511904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:56.511879 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj"] Oct 30 06:21:56.514108 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:21:56.514079 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8821eff_dc4e_456a_9112_5fee294c08ac.slice/crio-f6f9154cc133d0e3df5890e944f714f8b605e22cd594a9e1a6ea9b374438cd7b WatchSource:0}: Error finding container f6f9154cc133d0e3df5890e944f714f8b605e22cd594a9e1a6ea9b374438cd7b: Status 404 returned error can't find the container with id f6f9154cc133d0e3df5890e944f714f8b605e22cd594a9e1a6ea9b374438cd7b Oct 30 06:21:56.948557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:56.948522 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" event={"ID":"c8821eff-dc4e-456a-9112-5fee294c08ac","Type":"ContainerStarted","Data":"72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f"} Oct 30 06:21:56.948957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:56.948572 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" event={"ID":"c8821eff-dc4e-456a-9112-5fee294c08ac","Type":"ContainerStarted","Data":"f6f9154cc133d0e3df5890e944f714f8b605e22cd594a9e1a6ea9b374438cd7b"} Oct 30 06:21:56.948957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:56.948604 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:21:57.002046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:21:57.001996 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" podStartSLOduration=2.001981548 podStartE2EDuration="2.001981548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:56.988085258 +0000 UTC m=+1730.984042268" watchObservedRunningTime="2026-10-30 06:21:57.001981548 +0000 UTC m=+1730.997938557" Oct 30 06:22:07.959490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:07.959464 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:22:08.207641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:08.207609 5001 scope.go:122] "RemoveContainer" containerID="9b8301478f47b1219462d509b7813d2f5cfe1f2ce395bb376a6d1571b7d882c6" Oct 30 06:22:08.222352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:08.222322 5001 scope.go:122] "RemoveContainer" containerID="e09f04da5a808e1694a2a0b01c9b13d6005e72009e969e1ed347bf71675072eb" Oct 30 06:22:08.232997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:08.232977 5001 scope.go:122] "RemoveContainer" containerID="2c050295a0d893e8d10ea77c8c9da72e9b77614accf4ab9dd5d2a7359295acb7" Oct 30 06:22:08.275431 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:22:08.275398 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:22:17.773747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.773679 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c"] Oct 30 06:22:17.782817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.782797 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:17.814534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.814484 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-5a6371-openshift-default-dockercfg-rnlb4\"" Oct 30 06:22:17.829401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.825235 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c"] Oct 30 06:22:17.933962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.933924 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-credential-socket\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:17.934134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.933988 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-envoy\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:17.934134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.934015 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-socket\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:17.934134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.934037 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446c1aec-14d5-468e-887d-39c6579f389c-istio-podinfo\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:17.934394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.934266 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-data\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:17.934394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.934367 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446c1aec-14d5-468e-887d-39c6579f389c-istiod-ca-cert\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:17.934514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.934404 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-istio-token\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:17.934514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.934497 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br5jt\" (UniqueName: \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-kube-api-access-br5jt\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:17.934629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:17.934542 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-certs\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.039427 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-data\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.039509 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446c1aec-14d5-468e-887d-39c6579f389c-istiod-ca-cert\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.039547 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-istio-token\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.039588 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br5jt\" (UniqueName: \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-kube-api-access-br5jt\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.039631 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-certs\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.039736 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-credential-socket\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.039789 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-envoy\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.039822 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-socket\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.039853 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446c1aec-14d5-468e-887d-39c6579f389c-istio-podinfo\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.040890 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-credential-socket\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.041138 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-certs\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.047061 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446c1aec-14d5-468e-887d-39c6579f389c-istiod-ca-cert\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.049385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.047354 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-data\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.050659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.050630 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-socket\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.075755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.071021 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446c1aec-14d5-468e-887d-39c6579f389c-istio-podinfo\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.111412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.109591 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-envoy\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.145426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.136497 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-istio-token\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.169693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.168121 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br5jt\" (UniqueName: \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-kube-api-access-br5jt\") pod \"e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.428056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.428024 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:18.780867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.780836 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c"] Oct 30 06:22:18.783655 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:22:18.783622 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446c1aec_14d5_468e_887d_39c6579f389c.slice/crio-7952236c34dc1bce96f68e7fe11eceae26d58b4135f1efdbc450a7fcd68a6d5a WatchSource:0}: Error finding container 7952236c34dc1bce96f68e7fe11eceae26d58b4135f1efdbc450a7fcd68a6d5a: Status 404 returned error can't find the container with id 7952236c34dc1bce96f68e7fe11eceae26d58b4135f1efdbc450a7fcd68a6d5a Oct 30 06:22:18.787206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.787165 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:18.787339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.787258 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:18.787339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:18.787323 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:19.345140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:19.344843 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" event={"ID":"446c1aec-14d5-468e-887d-39c6579f389c","Type":"ContainerStarted","Data":"51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343"} Oct 30 06:22:19.345140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:19.344906 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" event={"ID":"446c1aec-14d5-468e-887d-39c6579f389c","Type":"ContainerStarted","Data":"7952236c34dc1bce96f68e7fe11eceae26d58b4135f1efdbc450a7fcd68a6d5a"} Oct 30 06:22:19.438356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:19.434173 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:19.473517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:19.462179 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:19.473517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:19.471626 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" podStartSLOduration=2.47160586 podStartE2EDuration="2.47160586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:19.409917979 +0000 UTC m=+1753.405874993" watchObservedRunningTime="2026-10-30 06:22:19.47160586 +0000 UTC m=+1753.467562871" Oct 30 06:22:20.358175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:20.358133 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:20.359602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:20.359577 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:22:21.649553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:21.649514 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:22:21.650064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:21.649588 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:22:23.347577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.347548 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:22:23.348016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.347605 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 06:22:23.444337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.441867 5001 generic.go:356] "Generic (PLEG): container finished" podID="92c83045-0841-4f12-a77c-ada2ae280906" containerID="5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460" exitCode=0 Oct 30 06:22:23.444337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.441982 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" event={"ID":"92c83045-0841-4f12-a77c-ada2ae280906","Type":"ContainerDied","Data":"5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460"} Oct 30 06:22:23.444337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.442555 5001 scope.go:122] "RemoveContainer" containerID="5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460" Oct 30 06:22:23.450576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.445180 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:23.450576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.445282 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:23.450576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.445357 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:23.471458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.471424 5001 generic.go:356] "Generic (PLEG): container finished" podID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerID="e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89" exitCode=0 Oct 30 06:22:23.471981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.471954 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" event={"ID":"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75","Type":"ContainerDied","Data":"e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89"} Oct 30 06:22:23.473110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.473091 5001 scope.go:122] "RemoveContainer" containerID="e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89" Oct 30 06:22:23.479610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.479561 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:23.479747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.479720 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:23.479843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.479826 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:23.808340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.804760 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4"] Oct 30 06:22:23.816331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.809016 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:22:23.816331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.811817 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4"] Oct 30 06:22:23.834670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.834632 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-5a6371\"" Oct 30 06:22:23.844142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.844115 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5"] Oct 30 06:22:23.848683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.848662 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:22:23.860192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.860165 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5"] Oct 30 06:22:23.863849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:23.863807 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-5a6371-dockercfg-mzxx9\"" Oct 30 06:22:24.091062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.083761 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87082fc2-7177-4533-bb11-3352c0a6bc66-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5\" (UID: \"87082fc2-7177-4533-bb11-3352c0a6bc66\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:22:24.091062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.083826 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb4pb\" (UniqueName: \"kubernetes.io/projected/87082fc2-7177-4533-bb11-3352c0a6bc66-kube-api-access-zb4pb\") pod \"payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5\" (UID: \"87082fc2-7177-4533-bb11-3352c0a6bc66\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:22:24.091062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.084063 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4\" (UID: \"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:22:24.091062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.084123 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp98z\" (UniqueName: \"kubernetes.io/projected/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-kube-api-access-wp98z\") pod \"payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4\" (UID: \"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:22:24.187330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.186434 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87082fc2-7177-4533-bb11-3352c0a6bc66-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5\" (UID: \"87082fc2-7177-4533-bb11-3352c0a6bc66\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:22:24.187330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.186489 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb4pb\" (UniqueName: \"kubernetes.io/projected/87082fc2-7177-4533-bb11-3352c0a6bc66-kube-api-access-zb4pb\") pod \"payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5\" (UID: \"87082fc2-7177-4533-bb11-3352c0a6bc66\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:22:24.187330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.186655 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4\" (UID: \"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:22:24.187330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.186709 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp98z\" (UniqueName: \"kubernetes.io/projected/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-kube-api-access-wp98z\") pod \"payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4\" (UID: \"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:22:24.191775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.191730 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87082fc2-7177-4533-bb11-3352c0a6bc66-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5\" (UID: \"87082fc2-7177-4533-bb11-3352c0a6bc66\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:22:24.193693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.193665 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4\" (UID: \"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:22:24.218257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.217578 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp98z\" (UniqueName: \"kubernetes.io/projected/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-kube-api-access-wp98z\") pod \"payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4\" (UID: \"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:22:24.221867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.221836 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb4pb\" (UniqueName: \"kubernetes.io/projected/87082fc2-7177-4533-bb11-3352c0a6bc66-kube-api-access-zb4pb\") pod \"payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5\" (UID: \"87082fc2-7177-4533-bb11-3352c0a6bc66\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:22:24.438090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.438049 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:22:24.488333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.487836 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:22:24.537852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.537819 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" event={"ID":"92c83045-0841-4f12-a77c-ada2ae280906","Type":"ContainerStarted","Data":"64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94"} Oct 30 06:22:24.541201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.541168 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" event={"ID":"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75","Type":"ContainerStarted","Data":"17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff"} Oct 30 06:22:24.627593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.623772 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:22:24.627593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.623882 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 06:22:24.627593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.623934 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 06:22:24.949298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.949268 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:22:24.949552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.949533 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:22:24.949641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:24.949578 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 06:22:25.046585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.046553 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5"] Oct 30 06:22:25.048023 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:22:25.047986 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87082fc2_7177_4533_bb11_3352c0a6bc66.slice/crio-4c46f0709348a3445fc6273115fd5f4a4bd87e319d726b2fb8d24edb517e5a3a WatchSource:0}: Error finding container 4c46f0709348a3445fc6273115fd5f4a4bd87e319d726b2fb8d24edb517e5a3a: Status 404 returned error can't find the container with id 4c46f0709348a3445fc6273115fd5f4a4bd87e319d726b2fb8d24edb517e5a3a Oct 30 06:22:25.111333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.095457 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4"] Oct 30 06:22:25.551392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.551263 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" event={"ID":"87082fc2-7177-4533-bb11-3352c0a6bc66","Type":"ContainerStarted","Data":"d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265"} Oct 30 06:22:25.551392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.551369 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" event={"ID":"87082fc2-7177-4533-bb11-3352c0a6bc66","Type":"ContainerStarted","Data":"4c46f0709348a3445fc6273115fd5f4a4bd87e319d726b2fb8d24edb517e5a3a"} Oct 30 06:22:25.551985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.551579 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:22:25.579330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.560649 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" event={"ID":"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde","Type":"ContainerStarted","Data":"aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677"} Oct 30 06:22:25.579609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.579578 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" event={"ID":"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde","Type":"ContainerStarted","Data":"4012c87cbc3da332237bf4234cf458f689f50ea6a0a5b6add4dfe40248926a44"} Oct 30 06:22:25.579757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.579742 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:22:25.619370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.619292 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 06:22:25.619562 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.619388 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 06:22:25.667470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.667414 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" podStartSLOduration=2.667392221 podStartE2EDuration="2.667392221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:25.612637984 +0000 UTC m=+1759.608594995" watchObservedRunningTime="2026-10-30 06:22:25.667392221 +0000 UTC m=+1759.663349232" Oct 30 06:22:25.668385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.668341 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" podStartSLOduration=2.668298368 podStartE2EDuration="2.668298368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:25.651574158 +0000 UTC m=+1759.647531186" watchObservedRunningTime="2026-10-30 06:22:25.668298368 +0000 UTC m=+1759.664255379" Oct 30 06:22:25.949586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.949548 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:22:25.949779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:25.949611 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 06:22:26.623957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:26.623919 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 06:22:26.624690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:26.624641 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 06:22:26.948932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:26.948841 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:22:26.948932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:26.948906 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 06:22:27.620098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:27.620059 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 06:22:27.620267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:27.620145 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 06:22:27.948810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:27.948727 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:22:27.949320 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:27.949280 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 06:22:28.619632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:28.619592 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 06:22:28.619809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:28.619662 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 06:22:28.948977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:28.948891 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:22:28.948977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:28.948948 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 06:22:29.629384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:29.624719 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 06:22:29.629384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:29.624798 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 06:22:29.949123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:29.949040 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:22:29.949123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:29.949095 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 06:22:30.624585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:30.624554 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:22:30.625073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:30.625059 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:22:30.625858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:30.625833 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:22:30.954085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:30.953997 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:22:30.954587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:30.954405 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:22:30.955293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:30.955268 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:22:36.640730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:36.640455 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:22:36.640730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:36.640521 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:22:38.277670 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:22:38.277630 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:22:38.483596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:38.483556 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:22:39.917284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:39.917248 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc"] Oct 30 06:22:39.921947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:39.921923 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:39.969171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:39.969134 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc"] Oct 30 06:22:39.992346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:39.992291 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:22:39.992652 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:39.992636 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:22:40.023586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.023537 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:22:40.040403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.037066 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:22:40.040630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.038211 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:40.040937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.040918 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:40.057657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.057611 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmsjr\" (UniqueName: \"kubernetes.io/projected/346d75fa-9f00-403e-840e-31e7949f83c3-kube-api-access-pmsjr\") pod \"maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:40.238398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.230675 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:40.238398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.230859 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmsjr\" (UniqueName: \"kubernetes.io/projected/346d75fa-9f00-403e-840e-31e7949f83c3-kube-api-access-pmsjr\") pod \"maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:40.238398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.230970 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:40.264373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.244195 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:40.305349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.292838 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:40.441330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.433131 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmsjr\" (UniqueName: \"kubernetes.io/projected/346d75fa-9f00-403e-840e-31e7949f83c3-kube-api-access-pmsjr\") pod \"maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:40.579386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:40.575525 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:41.413428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:41.413400 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc"] Oct 30 06:22:41.939606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:41.939568 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" event={"ID":"346d75fa-9f00-403e-840e-31e7949f83c3","Type":"ContainerStarted","Data":"a9b2269823475673928f7e710fd95b4c54f70f2e08aa33285978c7145da1725b"} Oct 30 06:22:42.977158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:42.977058 5001 generic.go:356] "Generic (PLEG): container finished" podID="346d75fa-9f00-403e-840e-31e7949f83c3" containerID="796ba6eb405daa951a903d74176acc18e535dd07276e41c88f94ad78b565ef09" exitCode=0 Oct 30 06:22:42.977880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:42.977851 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" event={"ID":"346d75fa-9f00-403e-840e-31e7949f83c3","Type":"ContainerDied","Data":"796ba6eb405daa951a903d74176acc18e535dd07276e41c88f94ad78b565ef09"} Oct 30 06:22:44.398623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:44.398241 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:44.487334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:44.468351 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-openshift-ca\") pod \"346d75fa-9f00-403e-840e-31e7949f83c3\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " Oct 30 06:22:44.487891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:44.487861 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346d75fa-9f00-403e-840e-31e7949f83c3-kube-api-access-pmsjr\" (UniqueName: \"kubernetes.io/projected/346d75fa-9f00-403e-840e-31e7949f83c3-kube-api-access-pmsjr\") pod \"346d75fa-9f00-403e-840e-31e7949f83c3\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " Oct 30 06:22:44.488186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:44.488163 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-xks-ca\") pod \"346d75fa-9f00-403e-840e-31e7949f83c3\" (UID: \"346d75fa-9f00-403e-840e-31e7949f83c3\") " Oct 30 06:22:44.501965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:44.501925 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-openshift-ca" pod "346d75fa-9f00-403e-840e-31e7949f83c3" (UID: "346d75fa-9f00-403e-840e-31e7949f83c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:44.511628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:44.508561 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346d75fa-9f00-403e-840e-31e7949f83c3-kube-api-access-pmsjr" pod "346d75fa-9f00-403e-840e-31e7949f83c3" (UID: "346d75fa-9f00-403e-840e-31e7949f83c3"). InnerVolumeSpecName "kube-api-access-pmsjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:44.511628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:44.511404 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-xks-ca" pod "346d75fa-9f00-403e-840e-31e7949f83c3" (UID: "346d75fa-9f00-403e-840e-31e7949f83c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:44.605999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:44.605952 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmsjr\" (UniqueName: \"kubernetes.io/projected/346d75fa-9f00-403e-840e-31e7949f83c3-kube-api-access-pmsjr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.605999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:44.606000 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.606267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:44.606017 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346d75fa-9f00-403e-840e-31e7949f83c3-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.023580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:45.023494 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" Oct 30 06:22:45.023580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:45.023532 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc" event={"ID":"346d75fa-9f00-403e-840e-31e7949f83c3","Type":"ContainerDied","Data":"a9b2269823475673928f7e710fd95b4c54f70f2e08aa33285978c7145da1725b"} Oct 30 06:22:45.023785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:45.023619 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b2269823475673928f7e710fd95b4c54f70f2e08aa33285978c7145da1725b" Oct 30 06:22:46.006345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:46.006282 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2_d923d506-2ebd-470d-b04d-ffa47379c56a/payload-processing/0.log" Oct 30 06:22:47.618318 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.618280 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l"] Oct 30 06:22:47.619217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.619197 5001 cpu_manager.go:414] "containerMap: removing container" podUID="346d75fa-9f00-403e-840e-31e7949f83c3" containerName="revoke-keys" Oct 30 06:22:47.619337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.619221 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="346d75fa-9f00-403e-840e-31e7949f83c3" containerName="revoke-keys" Oct 30 06:22:47.619436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.619423 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="346d75fa-9f00-403e-840e-31e7949f83c3" containerName="revoke-keys" Oct 30 06:22:47.623854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.623832 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:47.643266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.643233 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l"] Oct 30 06:22:47.655326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.655279 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:22:47.655507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.655408 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:22:47.655572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.655511 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:22:47.655651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.655587 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:22:47.687986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.687716 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:47.688277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.688253 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v5dw\" (UniqueName: \"kubernetes.io/projected/935f06ff-5347-4957-8d08-748c9d606a03-kube-api-access-7v5dw\") pod \"maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:47.688474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.688457 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:47.789952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.789907 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:47.790130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.790072 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:47.790130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.790106 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v5dw\" (UniqueName: \"kubernetes.io/projected/935f06ff-5347-4957-8d08-748c9d606a03-kube-api-access-7v5dw\") pod \"maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:47.790798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.790771 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:47.793299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.793269 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:47.818349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.814225 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v5dw\" (UniqueName: \"kubernetes.io/projected/935f06ff-5347-4957-8d08-748c9d606a03-kube-api-access-7v5dw\") pod \"maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:47.954533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:47.944154 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:48.331790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:48.331763 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l"] Oct 30 06:22:48.336347 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:22:48.336285 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935f06ff_5347_4957_8d08_748c9d606a03.slice/crio-d0d2a350454df918ecb715767c7605444dfdf1b06e3f6f023e30659ec7355902 WatchSource:0}: Error finding container d0d2a350454df918ecb715767c7605444dfdf1b06e3f6f023e30659ec7355902: Status 404 returned error can't find the container with id d0d2a350454df918ecb715767c7605444dfdf1b06e3f6f023e30659ec7355902 Oct 30 06:22:49.071664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:49.071562 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" event={"ID":"935f06ff-5347-4957-8d08-748c9d606a03","Type":"ContainerStarted","Data":"31a37f228bbba2d2ecc997e24aec2ae8a22db595a445a316e89f8d965ec84efd"} Oct 30 06:22:49.072175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:49.071663 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" event={"ID":"935f06ff-5347-4957-8d08-748c9d606a03","Type":"ContainerStarted","Data":"d0d2a350454df918ecb715767c7605444dfdf1b06e3f6f023e30659ec7355902"} Oct 30 06:22:49.111995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:49.111923 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" podStartSLOduration=2.111872213 podStartE2EDuration="2.111872213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:49.110599829 +0000 UTC m=+1783.106556841" watchObservedRunningTime="2026-10-30 06:22:49.111872213 +0000 UTC m=+1783.107829222" Oct 30 06:22:50.111381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:50.111145 5001 generic.go:356] "Generic (PLEG): container finished" podID="935f06ff-5347-4957-8d08-748c9d606a03" containerID="31a37f228bbba2d2ecc997e24aec2ae8a22db595a445a316e89f8d965ec84efd" exitCode=0 Oct 30 06:22:50.399502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:50.121451 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" event={"ID":"935f06ff-5347-4957-8d08-748c9d606a03","Type":"ContainerDied","Data":"31a37f228bbba2d2ecc997e24aec2ae8a22db595a445a316e89f8d965ec84efd"} Oct 30 06:22:51.029410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.029376 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc"] Oct 30 06:22:51.062376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.062291 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt"] Oct 30 06:22:51.127988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.127950 5001 generic.go:356] "Generic (PLEG): container finished" podID="448e441c-2924-4232-93c1-721a7b633880" containerID="56caec93d7705a6453a81c29e86f5f34c576b1d66eb32d979b7373c9088b8209" exitCode=0 Oct 30 06:22:51.128550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.128030 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" event={"ID":"448e441c-2924-4232-93c1-721a7b633880","Type":"ContainerDied","Data":"56caec93d7705a6453a81c29e86f5f34c576b1d66eb32d979b7373c9088b8209"} Oct 30 06:22:51.129023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.129000 5001 scope.go:122] "RemoveContainer" containerID="56caec93d7705a6453a81c29e86f5f34c576b1d66eb32d979b7373c9088b8209" Oct 30 06:22:51.131161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.131138 5001 generic.go:356] "Generic (PLEG): container finished" podID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerID="8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3" exitCode=0 Oct 30 06:22:51.131534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.131507 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" event={"ID":"eb19e5cd-434e-4d2d-bf69-ab0412742d3c","Type":"ContainerDied","Data":"8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3"} Oct 30 06:22:51.131956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.131931 5001 scope.go:122] "RemoveContainer" containerID="8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3" Oct 30 06:22:51.133693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.133647 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:51.133809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.133737 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:51.133809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.133786 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:51.134780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.134507 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:51.134780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.134590 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:51.134780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.134642 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:51.433497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.433414 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:51.524688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.524607 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-xks-ca\") pod \"935f06ff-5347-4957-8d08-748c9d606a03\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " Oct 30 06:22:51.525037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.525011 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935f06ff-5347-4957-8d08-748c9d606a03-kube-api-access-7v5dw\" (UniqueName: \"kubernetes.io/projected/935f06ff-5347-4957-8d08-748c9d606a03-kube-api-access-7v5dw\") pod \"935f06ff-5347-4957-8d08-748c9d606a03\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " Oct 30 06:22:51.525393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.525370 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-openshift-ca\") pod \"935f06ff-5347-4957-8d08-748c9d606a03\" (UID: \"935f06ff-5347-4957-8d08-748c9d606a03\") " Oct 30 06:22:51.526428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.526405 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-openshift-ca" pod "935f06ff-5347-4957-8d08-748c9d606a03" (UID: "935f06ff-5347-4957-8d08-748c9d606a03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:51.527522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.527496 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-xks-ca" pod "935f06ff-5347-4957-8d08-748c9d606a03" (UID: "935f06ff-5347-4957-8d08-748c9d606a03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:51.546049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.546017 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935f06ff-5347-4957-8d08-748c9d606a03-kube-api-access-7v5dw" pod "935f06ff-5347-4957-8d08-748c9d606a03" (UID: "935f06ff-5347-4957-8d08-748c9d606a03"). InnerVolumeSpecName "kube-api-access-7v5dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:51.627430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.627398 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:51.627430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.627427 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v5dw\" (UniqueName: \"kubernetes.io/projected/935f06ff-5347-4957-8d08-748c9d606a03-kube-api-access-7v5dw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:51.627430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.627437 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/935f06ff-5347-4957-8d08-748c9d606a03-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:51.902989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:51.891477 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:22:52.140879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:52.140854 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" Oct 30 06:22:52.141260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:52.140892 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l" event={"ID":"935f06ff-5347-4957-8d08-748c9d606a03","Type":"ContainerDied","Data":"d0d2a350454df918ecb715767c7605444dfdf1b06e3f6f023e30659ec7355902"} Oct 30 06:22:52.141260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:52.140940 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0d2a350454df918ecb715767c7605444dfdf1b06e3f6f023e30659ec7355902" Oct 30 06:22:52.143445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:52.143419 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" event={"ID":"448e441c-2924-4232-93c1-721a7b633880","Type":"ContainerStarted","Data":"a50db4a48de95522a3384399ca880d594f0e0fc31a303df4ec2ab70044afb63c"} Oct 30 06:22:52.145497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:52.145475 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" event={"ID":"eb19e5cd-434e-4d2d-bf69-ab0412742d3c","Type":"ContainerStarted","Data":"f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168"} Oct 30 06:22:52.583731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:52.583697 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:22:52.590499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:52.590457 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:22:52.590783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:52.590757 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:22:52.823539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:52.821810 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:22:52.823904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:52.823878 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:22:52.824111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:52.824086 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 06:22:53.579331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:53.578727 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:22:53.579331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:53.578793 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:22:53.827726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:53.827688 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:22:53.829288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:53.829255 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 06:22:53.902057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:53.902026 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc"] Oct 30 06:22:53.902556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:53.902523 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerName="istio-proxy" containerID="cri-o://feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5" gracePeriod=30 Oct 30 06:22:54.579329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:54.578727 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:22:54.579329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:54.578792 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:22:54.733837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:54.733801 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:22:54.829154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:54.829110 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:22:54.829511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:54.829441 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 06:22:55.591782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:55.591744 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:22:55.592455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:55.592418 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:22:55.825536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:55.825496 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:22:55.825826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:55.825797 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 06:22:56.030662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.030617 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" podUID="8d001f2c-4644-4fb5-a93f-0935613db130" containerName="payload-processing" containerID="cri-o://dadad052b248145b33c3f118e685d1ed6f5ac7597d39b215bc12f87f287a4a1e" gracePeriod=25 Oct 30 06:22:56.067425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.067375 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" podUID="d7cf442c-48ab-4054-8cb6-96680ce55215" containerName="payload-pre-processing" containerID="cri-o://92a4c626829624caff00ce477c6b5209eeabfb76ba1f15e01a27404c30f8ba02" gracePeriod=25 Oct 30 06:22:56.239963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.239927 5001 generic.go:356] "Generic (PLEG): container finished" podID="8d001f2c-4644-4fb5-a93f-0935613db130" containerID="dadad052b248145b33c3f118e685d1ed6f5ac7597d39b215bc12f87f287a4a1e" exitCode=0 Oct 30 06:22:56.240149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.240020 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" event={"ID":"8d001f2c-4644-4fb5-a93f-0935613db130","Type":"ContainerDied","Data":"dadad052b248145b33c3f118e685d1ed6f5ac7597d39b215bc12f87f287a4a1e"} Oct 30 06:22:56.242019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.241996 5001 generic.go:356] "Generic (PLEG): container finished" podID="d7cf442c-48ab-4054-8cb6-96680ce55215" containerID="92a4c626829624caff00ce477c6b5209eeabfb76ba1f15e01a27404c30f8ba02" exitCode=0 Oct 30 06:22:56.242130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.242085 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" event={"ID":"d7cf442c-48ab-4054-8cb6-96680ce55215","Type":"ContainerDied","Data":"92a4c626829624caff00ce477c6b5209eeabfb76ba1f15e01a27404c30f8ba02"} Oct 30 06:22:56.610884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.610765 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:22:56.610884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.610838 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:22:56.619519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.619466 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:22:56.638567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.637854 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d001f2c-4644-4fb5-a93f-0935613db130-kube-api-access-pfflk\" (UniqueName: \"kubernetes.io/projected/8d001f2c-4644-4fb5-a93f-0935613db130-kube-api-access-pfflk\") pod \"8d001f2c-4644-4fb5-a93f-0935613db130\" (UID: \"8d001f2c-4644-4fb5-a93f-0935613db130\") " Oct 30 06:22:56.638567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.637961 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d001f2c-4644-4fb5-a93f-0935613db130-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d001f2c-4644-4fb5-a93f-0935613db130-plugins-config-volume\") pod \"8d001f2c-4644-4fb5-a93f-0935613db130\" (UID: \"8d001f2c-4644-4fb5-a93f-0935613db130\") " Oct 30 06:22:56.639392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.639366 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d001f2c-4644-4fb5-a93f-0935613db130-plugins-config-volume" pod "8d001f2c-4644-4fb5-a93f-0935613db130" (UID: "8d001f2c-4644-4fb5-a93f-0935613db130"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:56.686341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.672743 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d001f2c-4644-4fb5-a93f-0935613db130-kube-api-access-pfflk" pod "8d001f2c-4644-4fb5-a93f-0935613db130" (UID: "8d001f2c-4644-4fb5-a93f-0935613db130"). InnerVolumeSpecName "kube-api-access-pfflk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:56.740343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.740081 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfflk\" (UniqueName: \"kubernetes.io/projected/8d001f2c-4644-4fb5-a93f-0935613db130-kube-api-access-pfflk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:56.740343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.740109 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d001f2c-4644-4fb5-a93f-0935613db130-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:56.749537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.749504 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2"] Oct 30 06:22:56.750226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.750204 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8d001f2c-4644-4fb5-a93f-0935613db130" containerName="payload-processing" Oct 30 06:22:56.750226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.750223 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d001f2c-4644-4fb5-a93f-0935613db130" containerName="payload-processing" Oct 30 06:22:56.750462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.750240 5001 cpu_manager.go:414] "containerMap: removing container" podUID="935f06ff-5347-4957-8d08-748c9d606a03" containerName="revoke-keys" Oct 30 06:22:56.750462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.750245 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935f06ff-5347-4957-8d08-748c9d606a03" containerName="revoke-keys" Oct 30 06:22:56.750462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.750458 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="935f06ff-5347-4957-8d08-748c9d606a03" containerName="revoke-keys" Oct 30 06:22:56.750618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.750488 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d001f2c-4644-4fb5-a93f-0935613db130" containerName="payload-processing" Oct 30 06:22:56.823331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.822534 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:22:56.823331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.822690 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 06:22:56.827367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.824638 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:22:56.827367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.826682 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2"] Oct 30 06:22:56.837571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.836673 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:22:56.841650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.839476 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:22:56.841650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.839708 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:22:56.841650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.840641 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:22:56.841650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.840735 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:22:56.841650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.840949 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:22:56.841650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.841172 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:22:56.841650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.841200 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skz87\" (UniqueName: \"kubernetes.io/projected/0347b7d3-1442-487a-ba4b-c787a6700116-kube-api-access-skz87\") pod \"maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:22:56.942502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.942418 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7cf442c-48ab-4054-8cb6-96680ce55215-kube-api-access-cf4s9\" (UniqueName: \"kubernetes.io/projected/d7cf442c-48ab-4054-8cb6-96680ce55215-kube-api-access-cf4s9\") pod \"d7cf442c-48ab-4054-8cb6-96680ce55215\" (UID: \"d7cf442c-48ab-4054-8cb6-96680ce55215\") " Oct 30 06:22:56.942642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.942502 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7cf442c-48ab-4054-8cb6-96680ce55215-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7cf442c-48ab-4054-8cb6-96680ce55215-plugins-config-volume\") pod \"d7cf442c-48ab-4054-8cb6-96680ce55215\" (UID: \"d7cf442c-48ab-4054-8cb6-96680ce55215\") " Oct 30 06:22:56.942902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.942875 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:22:56.943030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.942951 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7cf442c-48ab-4054-8cb6-96680ce55215-plugins-config-volume" pod "d7cf442c-48ab-4054-8cb6-96680ce55215" (UID: "d7cf442c-48ab-4054-8cb6-96680ce55215"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:56.943147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.943129 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:22:56.943199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.943162 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skz87\" (UniqueName: \"kubernetes.io/projected/0347b7d3-1442-487a-ba4b-c787a6700116-kube-api-access-skz87\") pod \"maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:22:56.943256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.943241 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7cf442c-48ab-4054-8cb6-96680ce55215-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:56.943626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.943601 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:22:56.944832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.944808 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cf442c-48ab-4054-8cb6-96680ce55215-kube-api-access-cf4s9" pod "d7cf442c-48ab-4054-8cb6-96680ce55215" (UID: "d7cf442c-48ab-4054-8cb6-96680ce55215"). InnerVolumeSpecName "kube-api-access-cf4s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:56.947951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:56.947927 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:22:57.016794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.016765 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skz87\" (UniqueName: \"kubernetes.io/projected/0347b7d3-1442-487a-ba4b-c787a6700116-kube-api-access-skz87\") pod \"maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:22:57.044786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.044755 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf4s9\" (UniqueName: \"kubernetes.io/projected/d7cf442c-48ab-4054-8cb6-96680ce55215-kube-api-access-cf4s9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:57.144917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.144883 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:22:57.326643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.322003 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" event={"ID":"8d001f2c-4644-4fb5-a93f-0935613db130","Type":"ContainerDied","Data":"c36be0784cec3939c16c3196e67e242c558bc3e8a5b1c5975c39ac7eaea88bd9"} Oct 30 06:22:57.326643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.322071 5001 scope.go:122] "RemoveContainer" containerID="dadad052b248145b33c3f118e685d1ed6f5ac7597d39b215bc12f87f287a4a1e" Oct 30 06:22:57.326643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.322238 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc" Oct 30 06:22:57.341836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.337177 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" Oct 30 06:22:57.341836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.337248 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt" event={"ID":"d7cf442c-48ab-4054-8cb6-96680ce55215","Type":"ContainerDied","Data":"0906675c88488ead4c4f7b669ca266c3ce525deda383739ded6f4f8c8ae15d5c"} Oct 30 06:22:57.349211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.345382 5001 scope.go:122] "RemoveContainer" containerID="92a4c626829624caff00ce477c6b5209eeabfb76ba1f15e01a27404c30f8ba02" Oct 30 06:22:57.522262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.521802 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc"] Oct 30 06:22:57.529049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.528988 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-520bc4-59bf859948-fn4xc"] Oct 30 06:22:57.571538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.556643 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2"] Oct 30 06:22:57.602374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.598743 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:22:57.602374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.598809 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:22:57.611835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.611759 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt"] Oct 30 06:22:57.614364 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:22:57.614297 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0347b7d3_1442_487a_ba4b_c787a6700116.slice/crio-65c0e149104aae7d0d16a63ec2ae31d5b35f69143491b30d2140070380201674 WatchSource:0}: Error finding container 65c0e149104aae7d0d16a63ec2ae31d5b35f69143491b30d2140070380201674: Status 404 returned error can't find the container with id 65c0e149104aae7d0d16a63ec2ae31d5b35f69143491b30d2140070380201674 Oct 30 06:22:57.616044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.615752 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-520bc4-6f768cffcf-86gjt"] Oct 30 06:22:57.825205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.821227 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:22:57.825205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:57.821282 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 06:22:58.121732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:58.121697 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d001f2c-4644-4fb5-a93f-0935613db130" path="/var/lib/kubelet/pods/8d001f2c-4644-4fb5-a93f-0935613db130/volumes" Oct 30 06:22:58.122374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:58.122353 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7cf442c-48ab-4054-8cb6-96680ce55215" path="/var/lib/kubelet/pods/d7cf442c-48ab-4054-8cb6-96680ce55215/volumes" Oct 30 06:22:58.348036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:58.347993 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" event={"ID":"0347b7d3-1442-487a-ba4b-c787a6700116","Type":"ContainerStarted","Data":"97bc3645e28f72d3811025d766214737bd7a418ffea5cc61206731545f866ac3"} Oct 30 06:22:58.348249 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:58.348093 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" event={"ID":"0347b7d3-1442-487a-ba4b-c787a6700116","Type":"ContainerStarted","Data":"65c0e149104aae7d0d16a63ec2ae31d5b35f69143491b30d2140070380201674"} Oct 30 06:22:58.443975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:58.443854 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" podStartSLOduration=2.443838352 podStartE2EDuration="2.443838352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:58.442474761 +0000 UTC m=+1792.438431772" watchObservedRunningTime="2026-10-30 06:22:58.443838352 +0000 UTC m=+1792.439795361" Oct 30 06:22:58.582010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:58.581975 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:22:58.582557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:58.582541 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:22:58.583339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:58.583291 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:22:58.826909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:58.826881 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:22:58.827996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:58.827978 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:22:59.219045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.219021 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:22:59.318107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.318007 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-token\" (UniqueName: \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-token\") pod \"74df9a78-380f-4540-a6b5-8bfefb73a099\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " Oct 30 06:22:59.318315 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.318124 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-credential-socket\") pod \"74df9a78-380f-4540-a6b5-8bfefb73a099\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " Oct 30 06:22:59.318462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.318437 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-credential-socket" pod "74df9a78-380f-4540-a6b5-8bfefb73a099" (UID: "74df9a78-380f-4540-a6b5-8bfefb73a099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:59.318516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.318494 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-socket\") pod \"74df9a78-380f-4540-a6b5-8bfefb73a099\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " Oct 30 06:22:59.318623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.318607 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-podinfo\") pod \"74df9a78-380f-4540-a6b5-8bfefb73a099\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " Oct 30 06:22:59.318703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.318674 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74df9a78-380f-4540-a6b5-8bfefb73a099-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74df9a78-380f-4540-a6b5-8bfefb73a099-istiod-ca-cert\") pod \"74df9a78-380f-4540-a6b5-8bfefb73a099\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " Oct 30 06:22:59.318785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.318762 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-data\") pod \"74df9a78-380f-4540-a6b5-8bfefb73a099\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " Oct 30 06:22:59.318951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.318828 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-envoy\") pod \"74df9a78-380f-4540-a6b5-8bfefb73a099\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " Oct 30 06:22:59.318951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.318895 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-certs\") pod \"74df9a78-380f-4540-a6b5-8bfefb73a099\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " Oct 30 06:22:59.318951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.318944 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-kube-api-access-j97zc\" (UniqueName: \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-kube-api-access-j97zc\") pod \"74df9a78-380f-4540-a6b5-8bfefb73a099\" (UID: \"74df9a78-380f-4540-a6b5-8bfefb73a099\") " Oct 30 06:22:59.319797 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.319774 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:59.320515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.320460 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-data" pod "74df9a78-380f-4540-a6b5-8bfefb73a099" (UID: "74df9a78-380f-4540-a6b5-8bfefb73a099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:59.320636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.320576 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-socket" pod "74df9a78-380f-4540-a6b5-8bfefb73a099" (UID: "74df9a78-380f-4540-a6b5-8bfefb73a099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:59.320817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.320740 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74df9a78-380f-4540-a6b5-8bfefb73a099-istiod-ca-cert" pod "74df9a78-380f-4540-a6b5-8bfefb73a099" (UID: "74df9a78-380f-4540-a6b5-8bfefb73a099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:59.320817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.320790 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-certs" pod "74df9a78-380f-4540-a6b5-8bfefb73a099" (UID: "74df9a78-380f-4540-a6b5-8bfefb73a099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:59.321987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.321962 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-podinfo" pod "74df9a78-380f-4540-a6b5-8bfefb73a099" (UID: "74df9a78-380f-4540-a6b5-8bfefb73a099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:59.322128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.321964 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-token" pod "74df9a78-380f-4540-a6b5-8bfefb73a099" (UID: "74df9a78-380f-4540-a6b5-8bfefb73a099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:59.322839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.322805 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-envoy" pod "74df9a78-380f-4540-a6b5-8bfefb73a099" (UID: "74df9a78-380f-4540-a6b5-8bfefb73a099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:59.323270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.323237 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-kube-api-access-j97zc" pod "74df9a78-380f-4540-a6b5-8bfefb73a099" (UID: "74df9a78-380f-4540-a6b5-8bfefb73a099"). InnerVolumeSpecName "kube-api-access-j97zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:59.361206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.361173 5001 generic.go:356] "Generic (PLEG): container finished" podID="0347b7d3-1442-487a-ba4b-c787a6700116" containerID="97bc3645e28f72d3811025d766214737bd7a418ffea5cc61206731545f866ac3" exitCode=0 Oct 30 06:22:59.361364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.361258 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" event={"ID":"0347b7d3-1442-487a-ba4b-c787a6700116","Type":"ContainerDied","Data":"97bc3645e28f72d3811025d766214737bd7a418ffea5cc61206731545f866ac3"} Oct 30 06:22:59.362688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.362664 5001 generic.go:356] "Generic (PLEG): container finished" podID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerID="feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5" exitCode=0 Oct 30 06:22:59.362793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.362719 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" Oct 30 06:22:59.362793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.362754 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" event={"ID":"74df9a78-380f-4540-a6b5-8bfefb73a099","Type":"ContainerDied","Data":"feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5"} Oct 30 06:22:59.362914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.362804 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc" event={"ID":"74df9a78-380f-4540-a6b5-8bfefb73a099","Type":"ContainerDied","Data":"edcffcd65124f3b61d613b1054cd18919e63d5289c6a49bf3ffe3c73d27acf16"} Oct 30 06:22:59.362914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.362826 5001 scope.go:122] "RemoveContainer" containerID="feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5" Oct 30 06:22:59.394271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.393510 5001 scope.go:122] "RemoveContainer" containerID="feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5" Oct 30 06:22:59.394271 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:22:59.393993 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5\": container with ID starting with feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5 not found: ID does not exist" containerID="feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5" Oct 30 06:22:59.394271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.394030 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5"} err="failed to get container status \"feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5\": rpc error: code = NotFound desc = could not find container \"feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5\": container with ID starting with feb600693520f14a7a4b71d2fb33ff0e064db066afe205a9d800bdf6f3e155c5 not found: ID does not exist" Oct 30 06:22:59.421491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.421459 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:59.421491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.421493 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:59.421752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.421507 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:59.421752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.421522 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j97zc\" (UniqueName: \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-kube-api-access-j97zc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:59.421752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.421540 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:59.421752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.421554 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74df9a78-380f-4540-a6b5-8bfefb73a099-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:59.421752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.421567 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74df9a78-380f-4540-a6b5-8bfefb73a099-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:59.421752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.421581 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74df9a78-380f-4540-a6b5-8bfefb73a099-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:22:59.441371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.441341 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc"] Oct 30 06:22:59.445596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:22:59.445571 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-hb4pc"] Oct 30 06:23:00.119119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:00.119090 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" path="/var/lib/kubelet/pods/74df9a78-380f-4540-a6b5-8bfefb73a099/volumes" Oct 30 06:23:00.541215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:00.541185 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:23:00.638590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:00.638509 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0347b7d3-1442-487a-ba4b-c787a6700116-kube-api-access-skz87\" (UniqueName: \"kubernetes.io/projected/0347b7d3-1442-487a-ba4b-c787a6700116-kube-api-access-skz87\") pod \"0347b7d3-1442-487a-ba4b-c787a6700116\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " Oct 30 06:23:00.638590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:00.638556 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-xks-ca\") pod \"0347b7d3-1442-487a-ba4b-c787a6700116\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " Oct 30 06:23:00.638843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:00.638737 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-openshift-ca\") pod \"0347b7d3-1442-487a-ba4b-c787a6700116\" (UID: \"0347b7d3-1442-487a-ba4b-c787a6700116\") " Oct 30 06:23:00.639094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:00.639069 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-openshift-ca" pod "0347b7d3-1442-487a-ba4b-c787a6700116" (UID: "0347b7d3-1442-487a-ba4b-c787a6700116"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:00.640781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:00.640754 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0347b7d3-1442-487a-ba4b-c787a6700116-kube-api-access-skz87" pod "0347b7d3-1442-487a-ba4b-c787a6700116" (UID: "0347b7d3-1442-487a-ba4b-c787a6700116"). InnerVolumeSpecName "kube-api-access-skz87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:00.640865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:00.640825 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-xks-ca" pod "0347b7d3-1442-487a-ba4b-c787a6700116" (UID: "0347b7d3-1442-487a-ba4b-c787a6700116"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:00.741015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:00.740979 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:00.741015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:00.741010 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skz87\" (UniqueName: \"kubernetes.io/projected/0347b7d3-1442-487a-ba4b-c787a6700116-kube-api-access-skz87\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:00.741015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:00.741019 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0347b7d3-1442-487a-ba4b-c787a6700116-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.383802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:01.383769 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" Oct 30 06:23:01.383802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:01.383780 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2" event={"ID":"0347b7d3-1442-487a-ba4b-c787a6700116","Type":"ContainerDied","Data":"65c0e149104aae7d0d16a63ec2ae31d5b35f69143491b30d2140070380201674"} Oct 30 06:23:01.384252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:01.383829 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65c0e149104aae7d0d16a63ec2ae31d5b35f69143491b30d2140070380201674" Oct 30 06:23:05.320193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.320156 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97"] Oct 30 06:23:05.320853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.320836 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0347b7d3-1442-487a-ba4b-c787a6700116" containerName="revoke-keys" Oct 30 06:23:05.320899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.320854 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0347b7d3-1442-487a-ba4b-c787a6700116" containerName="revoke-keys" Oct 30 06:23:05.320899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.320869 5001 cpu_manager.go:414] "containerMap: removing container" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerName="istio-proxy" Oct 30 06:23:05.320899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.320874 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerName="istio-proxy" Oct 30 06:23:05.321053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.320909 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d7cf442c-48ab-4054-8cb6-96680ce55215" containerName="payload-pre-processing" Oct 30 06:23:05.321053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.320915 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7cf442c-48ab-4054-8cb6-96680ce55215" containerName="payload-pre-processing" Oct 30 06:23:05.321053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.321036 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7cf442c-48ab-4054-8cb6-96680ce55215" containerName="payload-pre-processing" Oct 30 06:23:05.321053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.321047 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0347b7d3-1442-487a-ba4b-c787a6700116" containerName="revoke-keys" Oct 30 06:23:05.321178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.321056 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="74df9a78-380f-4540-a6b5-8bfefb73a099" containerName="istio-proxy" Oct 30 06:23:05.323749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.323734 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:05.345643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.345617 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97"] Oct 30 06:23:05.352427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.352400 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:23:05.352574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.352454 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:05.352574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.352468 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:23:05.352574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.352539 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:23:05.520647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.520604 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2fdd6fc-mnc97\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:05.520647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.520647 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2fdd6fc-mnc97\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:05.520918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.520898 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2m4\" (UniqueName: \"kubernetes.io/projected/3450d685-8949-4d2f-a820-bbc371682fc2-kube-api-access-nd2m4\") pod \"maas-api-revoke-sub-b2fdd6fc-mnc97\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:05.621699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.621670 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2m4\" (UniqueName: \"kubernetes.io/projected/3450d685-8949-4d2f-a820-bbc371682fc2-kube-api-access-nd2m4\") pod \"maas-api-revoke-sub-b2fdd6fc-mnc97\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:05.621899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.621772 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2fdd6fc-mnc97\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:05.621899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.621791 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2fdd6fc-mnc97\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:05.622475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.622452 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2fdd6fc-mnc97\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:05.624257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.624238 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2fdd6fc-mnc97\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:05.646708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.646667 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2m4\" (UniqueName: \"kubernetes.io/projected/3450d685-8949-4d2f-a820-bbc371682fc2-kube-api-access-nd2m4\") pod \"maas-api-revoke-sub-b2fdd6fc-mnc97\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:05.937117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:05.937029 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:06.255660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:06.255600 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97"] Oct 30 06:23:06.259235 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:23:06.259192 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3450d685_8949_4d2f_a820_bbc371682fc2.slice/crio-5c10b2f75791dce9aa6f4bbf7b2211dd3aa2d6def5209d06de71e851f62b8570 WatchSource:0}: Error finding container 5c10b2f75791dce9aa6f4bbf7b2211dd3aa2d6def5209d06de71e851f62b8570: Status 404 returned error can't find the container with id 5c10b2f75791dce9aa6f4bbf7b2211dd3aa2d6def5209d06de71e851f62b8570 Oct 30 06:23:06.435125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:06.435090 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" event={"ID":"3450d685-8949-4d2f-a820-bbc371682fc2","Type":"ContainerStarted","Data":"5c10b2f75791dce9aa6f4bbf7b2211dd3aa2d6def5209d06de71e851f62b8570"} Oct 30 06:23:06.516361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:06.516253 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:23:06.532360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:06.532334 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:23:07.215316 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:07.215267 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k"] Oct 30 06:23:07.280905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:07.280425 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl"] Oct 30 06:23:07.445833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:07.445801 5001 generic.go:356] "Generic (PLEG): container finished" podID="3450d685-8949-4d2f-a820-bbc371682fc2" containerID="63b3d982306c19da3a7f17053185605c8adeca00c633e27506beaa739c88b254" exitCode=0 Oct 30 06:23:07.446187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:07.445840 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" event={"ID":"3450d685-8949-4d2f-a820-bbc371682fc2","Type":"ContainerDied","Data":"63b3d982306c19da3a7f17053185605c8adeca00c633e27506beaa739c88b254"} Oct 30 06:23:08.276541 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:23:08.276497 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:23:08.316330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.313820 5001 scope.go:122] "RemoveContainer" containerID="f9f092ce75f21928c6ddb7b0a3c24b325249e6911f891ef23d00c29462c6d081" Oct 30 06:23:08.336473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.336448 5001 scope.go:122] "RemoveContainer" containerID="82374251e9b5ebb1a23cd2225a81bf478381fc1f5156b5227fc75a1417aea781" Oct 30 06:23:08.726660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.726634 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:08.895926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.895881 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-xks-ca\") pod \"3450d685-8949-4d2f-a820-bbc371682fc2\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " Oct 30 06:23:08.896087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.896038 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3450d685-8949-4d2f-a820-bbc371682fc2-kube-api-access-nd2m4\" (UniqueName: \"kubernetes.io/projected/3450d685-8949-4d2f-a820-bbc371682fc2-kube-api-access-nd2m4\") pod \"3450d685-8949-4d2f-a820-bbc371682fc2\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " Oct 30 06:23:08.896159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.896104 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-openshift-ca\") pod \"3450d685-8949-4d2f-a820-bbc371682fc2\" (UID: \"3450d685-8949-4d2f-a820-bbc371682fc2\") " Oct 30 06:23:08.896623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.896509 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-openshift-ca" pod "3450d685-8949-4d2f-a820-bbc371682fc2" (UID: "3450d685-8949-4d2f-a820-bbc371682fc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:08.896776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.896740 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.898913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.898887 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-xks-ca" pod "3450d685-8949-4d2f-a820-bbc371682fc2" (UID: "3450d685-8949-4d2f-a820-bbc371682fc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:08.899097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.899062 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3450d685-8949-4d2f-a820-bbc371682fc2-kube-api-access-nd2m4" pod "3450d685-8949-4d2f-a820-bbc371682fc2" (UID: "3450d685-8949-4d2f-a820-bbc371682fc2"). InnerVolumeSpecName "kube-api-access-nd2m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:08.997464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.997433 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3450d685-8949-4d2f-a820-bbc371682fc2-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.997464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:08.997461 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd2m4\" (UniqueName: \"kubernetes.io/projected/3450d685-8949-4d2f-a820-bbc371682fc2-kube-api-access-nd2m4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:09.444487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:09.444448 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2"] Oct 30 06:23:09.500326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:09.490549 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-bbd0e5-nnlg2"] Oct 30 06:23:09.521506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:09.521471 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" event={"ID":"3450d685-8949-4d2f-a820-bbc371682fc2","Type":"ContainerDied","Data":"5c10b2f75791dce9aa6f4bbf7b2211dd3aa2d6def5209d06de71e851f62b8570"} Oct 30 06:23:09.521697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:09.521512 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97" Oct 30 06:23:09.521697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:09.521530 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c10b2f75791dce9aa6f4bbf7b2211dd3aa2d6def5209d06de71e851f62b8570" Oct 30 06:23:10.119001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:10.118969 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0347b7d3-1442-487a-ba4b-c787a6700116" path="/var/lib/kubelet/pods/0347b7d3-1442-487a-ba4b-c787a6700116/volumes" Oct 30 06:23:12.216113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.216068 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" podUID="a2ec7a66-b879-4514-9ba2-3249d56ae8ad" containerName="payload-processing" containerID="cri-o://889bac59709ef3db1ba42b094eea07ccab5b54bdd565b78f8b24da9288278793" gracePeriod=25 Oct 30 06:23:12.282468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.282421 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" podUID="d4324bad-9424-4246-8b75-5e6e2a208739" containerName="payload-pre-processing" containerID="cri-o://b04ea33c7da34d5134f301448dbc09ea0c8b4772b4b9972f36e979abb2d571c9" gracePeriod=25 Oct 30 06:23:12.614882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.612061 5001 generic.go:356] "Generic (PLEG): container finished" podID="a2ec7a66-b879-4514-9ba2-3249d56ae8ad" containerID="889bac59709ef3db1ba42b094eea07ccab5b54bdd565b78f8b24da9288278793" exitCode=0 Oct 30 06:23:12.614882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.612215 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" event={"ID":"a2ec7a66-b879-4514-9ba2-3249d56ae8ad","Type":"ContainerDied","Data":"889bac59709ef3db1ba42b094eea07ccab5b54bdd565b78f8b24da9288278793"} Oct 30 06:23:12.620487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.620046 5001 generic.go:356] "Generic (PLEG): container finished" podID="d4324bad-9424-4246-8b75-5e6e2a208739" containerID="b04ea33c7da34d5134f301448dbc09ea0c8b4772b4b9972f36e979abb2d571c9" exitCode=0 Oct 30 06:23:12.620487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.620169 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" event={"ID":"d4324bad-9424-4246-8b75-5e6e2a208739","Type":"ContainerDied","Data":"b04ea33c7da34d5134f301448dbc09ea0c8b4772b4b9972f36e979abb2d571c9"} Oct 30 06:23:12.709958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.709929 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:23:12.714152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.714134 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:23:12.788910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.788821 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4324bad-9424-4246-8b75-5e6e2a208739-kube-api-access-99tgq\" (UniqueName: \"kubernetes.io/projected/d4324bad-9424-4246-8b75-5e6e2a208739-kube-api-access-99tgq\") pod \"d4324bad-9424-4246-8b75-5e6e2a208739\" (UID: \"d4324bad-9424-4246-8b75-5e6e2a208739\") " Oct 30 06:23:12.789882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.789859 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-plugins-config-volume\") pod \"a2ec7a66-b879-4514-9ba2-3249d56ae8ad\" (UID: \"a2ec7a66-b879-4514-9ba2-3249d56ae8ad\") " Oct 30 06:23:12.790611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.790414 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-plugins-config-volume" pod "a2ec7a66-b879-4514-9ba2-3249d56ae8ad" (UID: "a2ec7a66-b879-4514-9ba2-3249d56ae8ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:12.790611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.790515 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4324bad-9424-4246-8b75-5e6e2a208739-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4324bad-9424-4246-8b75-5e6e2a208739-plugins-config-volume\") pod \"d4324bad-9424-4246-8b75-5e6e2a208739\" (UID: \"d4324bad-9424-4246-8b75-5e6e2a208739\") " Oct 30 06:23:12.791070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.790909 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-kube-api-access-4xnck\" (UniqueName: \"kubernetes.io/projected/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-kube-api-access-4xnck\") pod \"a2ec7a66-b879-4514-9ba2-3249d56ae8ad\" (UID: \"a2ec7a66-b879-4514-9ba2-3249d56ae8ad\") " Oct 30 06:23:12.791796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.791607 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4324bad-9424-4246-8b75-5e6e2a208739-plugins-config-volume" pod "d4324bad-9424-4246-8b75-5e6e2a208739" (UID: "d4324bad-9424-4246-8b75-5e6e2a208739"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:12.791954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.791896 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:12.792212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.792195 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4324bad-9424-4246-8b75-5e6e2a208739-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:12.794897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.794858 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4324bad-9424-4246-8b75-5e6e2a208739-kube-api-access-99tgq" pod "d4324bad-9424-4246-8b75-5e6e2a208739" (UID: "d4324bad-9424-4246-8b75-5e6e2a208739"). InnerVolumeSpecName "kube-api-access-99tgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:12.796753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.796728 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-kube-api-access-4xnck" pod "a2ec7a66-b879-4514-9ba2-3249d56ae8ad" (UID: "a2ec7a66-b879-4514-9ba2-3249d56ae8ad"). InnerVolumeSpecName "kube-api-access-4xnck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:12.894440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.894405 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xnck\" (UniqueName: \"kubernetes.io/projected/a2ec7a66-b879-4514-9ba2-3249d56ae8ad-kube-api-access-4xnck\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:12.894440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:12.894436 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99tgq\" (UniqueName: \"kubernetes.io/projected/d4324bad-9424-4246-8b75-5e6e2a208739-kube-api-access-99tgq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:13.269382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:13.253093 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:23:13.679334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:13.679258 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" event={"ID":"a2ec7a66-b879-4514-9ba2-3249d56ae8ad","Type":"ContainerDied","Data":"0f349d2f3137db6b783da3fa4b3a2b97b3c027826f3a440b0e9088e5877d1ebe"} Oct 30 06:23:13.679524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:13.679349 5001 scope.go:122] "RemoveContainer" containerID="889bac59709ef3db1ba42b094eea07ccab5b54bdd565b78f8b24da9288278793" Oct 30 06:23:13.679524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:13.679503 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k" Oct 30 06:23:13.682791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:13.682755 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" event={"ID":"d4324bad-9424-4246-8b75-5e6e2a208739","Type":"ContainerDied","Data":"c451f187df51fd178eee4089dad4d28768bdcdb06c6a1ea9c55c93f4f9463c52"} Oct 30 06:23:13.683070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:13.683054 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl" Oct 30 06:23:13.739285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:13.739258 5001 scope.go:122] "RemoveContainer" containerID="b04ea33c7da34d5134f301448dbc09ea0c8b4772b4b9972f36e979abb2d571c9" Oct 30 06:23:13.831640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:13.829607 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl"] Oct 30 06:23:13.867849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:13.867810 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-bbd0e5-77945bd86c-p2rcl"] Oct 30 06:23:13.892797 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:13.892728 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k"] Oct 30 06:23:13.903984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:13.903922 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-bbd0e5-64d48998dd-ngp6k"] Oct 30 06:23:14.033629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:14.033522 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc"] Oct 30 06:23:14.040056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:14.040011 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-520bc4-gwrkc"] Oct 30 06:23:14.169348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:14.165880 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="346d75fa-9f00-403e-840e-31e7949f83c3" path="/var/lib/kubelet/pods/346d75fa-9f00-403e-840e-31e7949f83c3/volumes" Oct 30 06:23:14.184297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:14.183530 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ec7a66-b879-4514-9ba2-3249d56ae8ad" path="/var/lib/kubelet/pods/a2ec7a66-b879-4514-9ba2-3249d56ae8ad/volumes" Oct 30 06:23:14.184297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:14.184222 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4324bad-9424-4246-8b75-5e6e2a208739" path="/var/lib/kubelet/pods/d4324bad-9424-4246-8b75-5e6e2a208739/volumes" Oct 30 06:23:15.836494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:15.836458 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:23:15.836901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:15.836522 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" podUID="655aebda-9cbb-4aae-b560-ee17928630b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:23:18.009392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.009348 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn"] Oct 30 06:23:18.010525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.010490 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" podUID="655aebda-9cbb-4aae-b560-ee17928630b9" containerName="istio-proxy" containerID="cri-o://7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8" gracePeriod=30 Oct 30 06:23:18.485327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.485273 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24"] Oct 30 06:23:18.486407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.486384 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d4324bad-9424-4246-8b75-5e6e2a208739" containerName="payload-pre-processing" Oct 30 06:23:18.486407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.486408 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4324bad-9424-4246-8b75-5e6e2a208739" containerName="payload-pre-processing" Oct 30 06:23:18.486560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.486477 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a2ec7a66-b879-4514-9ba2-3249d56ae8ad" containerName="payload-processing" Oct 30 06:23:18.486560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.486487 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ec7a66-b879-4514-9ba2-3249d56ae8ad" containerName="payload-processing" Oct 30 06:23:18.486560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.486508 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3450d685-8949-4d2f-a820-bbc371682fc2" containerName="revoke-keys" Oct 30 06:23:18.486560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.486516 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3450d685-8949-4d2f-a820-bbc371682fc2" containerName="revoke-keys" Oct 30 06:23:18.486691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.486676 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4324bad-9424-4246-8b75-5e6e2a208739" containerName="payload-pre-processing" Oct 30 06:23:18.486730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.486707 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ec7a66-b879-4514-9ba2-3249d56ae8ad" containerName="payload-processing" Oct 30 06:23:18.486730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.486720 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3450d685-8949-4d2f-a820-bbc371682fc2" containerName="revoke-keys" Oct 30 06:23:18.493789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.493766 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:18.502753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.502574 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24"] Oct 30 06:23:18.515362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.515334 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:23:18.515952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.515908 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk755\" (UniqueName: \"kubernetes.io/projected/79f0d6ea-d80a-4c14-8135-9647816f7eff-kube-api-access-sk755\") pod \"maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:18.516056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.515976 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:18.516056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.515991 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:23:18.516186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.516085 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:18.534672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.534642 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:18.534819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.534684 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:23:18.617733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.617696 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk755\" (UniqueName: \"kubernetes.io/projected/79f0d6ea-d80a-4c14-8135-9647816f7eff-kube-api-access-sk755\") pod \"maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:18.617906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.617744 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:18.617906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.617805 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:18.618582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.618555 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:18.620400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.620378 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:18.641132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.641099 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk755\" (UniqueName: \"kubernetes.io/projected/79f0d6ea-d80a-4c14-8135-9647816f7eff-kube-api-access-sk755\") pod \"maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:18.817472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:18.817390 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:19.171359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:19.167708 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24"] Oct 30 06:23:19.183861 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:23:19.183826 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f0d6ea_d80a_4c14_8135_9647816f7eff.slice/crio-f51cd2ccba28322d6a40066ec0b5cfb490ad817ddaea296c3faba4b3021e4aca WatchSource:0}: Error finding container f51cd2ccba28322d6a40066ec0b5cfb490ad817ddaea296c3faba4b3021e4aca: Status 404 returned error can't find the container with id f51cd2ccba28322d6a40066ec0b5cfb490ad817ddaea296c3faba4b3021e4aca Oct 30 06:23:19.833384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:19.833276 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" event={"ID":"79f0d6ea-d80a-4c14-8135-9647816f7eff","Type":"ContainerStarted","Data":"899a50b95f0e71d909e1f8cf463c3045883f4231f2a562a2e893b2dfde745c91"} Oct 30 06:23:19.833384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:19.833378 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" event={"ID":"79f0d6ea-d80a-4c14-8135-9647816f7eff","Type":"ContainerStarted","Data":"f51cd2ccba28322d6a40066ec0b5cfb490ad817ddaea296c3faba4b3021e4aca"} Oct 30 06:23:20.657689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.657656 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:23:20.729346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.729261 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" podStartSLOduration=2.729241661 podStartE2EDuration="2.729241661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:19.88176224 +0000 UTC m=+1813.877719257" watchObservedRunningTime="2026-10-30 06:23:20.729241661 +0000 UTC m=+1814.725198672" Oct 30 06:23:20.748020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.747986 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-socket\") pod \"655aebda-9cbb-4aae-b560-ee17928630b9\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " Oct 30 06:23:20.748215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.748049 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-istio-token\" (UniqueName: \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-istio-token\") pod \"655aebda-9cbb-4aae-b560-ee17928630b9\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " Oct 30 06:23:20.748215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.748126 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-kube-api-access-8l6nk\" (UniqueName: \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-kube-api-access-8l6nk\") pod \"655aebda-9cbb-4aae-b560-ee17928630b9\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " Oct 30 06:23:20.748215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.748162 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655aebda-9cbb-4aae-b560-ee17928630b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655aebda-9cbb-4aae-b560-ee17928630b9-istiod-ca-cert\") pod \"655aebda-9cbb-4aae-b560-ee17928630b9\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " Oct 30 06:23:20.748215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.748193 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-credential-socket\") pod \"655aebda-9cbb-4aae-b560-ee17928630b9\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " Oct 30 06:23:20.748474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.748451 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-socket" pod "655aebda-9cbb-4aae-b560-ee17928630b9" (UID: "655aebda-9cbb-4aae-b560-ee17928630b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.748942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.748814 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655aebda-9cbb-4aae-b560-ee17928630b9-istiod-ca-cert" pod "655aebda-9cbb-4aae-b560-ee17928630b9" (UID: "655aebda-9cbb-4aae-b560-ee17928630b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:20.749124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.749106 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-credential-socket" pod "655aebda-9cbb-4aae-b560-ee17928630b9" (UID: "655aebda-9cbb-4aae-b560-ee17928630b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.749687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.749580 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-envoy\") pod \"655aebda-9cbb-4aae-b560-ee17928630b9\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " Oct 30 06:23:20.749687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.749614 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/655aebda-9cbb-4aae-b560-ee17928630b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655aebda-9cbb-4aae-b560-ee17928630b9-istio-podinfo\") pod \"655aebda-9cbb-4aae-b560-ee17928630b9\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " Oct 30 06:23:20.749687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.749645 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-data\") pod \"655aebda-9cbb-4aae-b560-ee17928630b9\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " Oct 30 06:23:20.749986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.749844 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-certs\") pod \"655aebda-9cbb-4aae-b560-ee17928630b9\" (UID: \"655aebda-9cbb-4aae-b560-ee17928630b9\") " Oct 30 06:23:20.750639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.750461 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.750639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.750487 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655aebda-9cbb-4aae-b560-ee17928630b9-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.750639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.750490 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-data" pod "655aebda-9cbb-4aae-b560-ee17928630b9" (UID: "655aebda-9cbb-4aae-b560-ee17928630b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.750639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.750502 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.750639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.750598 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-certs" pod "655aebda-9cbb-4aae-b560-ee17928630b9" (UID: "655aebda-9cbb-4aae-b560-ee17928630b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.754666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.754625 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-kube-api-access-8l6nk" pod "655aebda-9cbb-4aae-b560-ee17928630b9" (UID: "655aebda-9cbb-4aae-b560-ee17928630b9"). InnerVolumeSpecName "kube-api-access-8l6nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:20.755210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.755172 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/655aebda-9cbb-4aae-b560-ee17928630b9-istio-podinfo" pod "655aebda-9cbb-4aae-b560-ee17928630b9" (UID: "655aebda-9cbb-4aae-b560-ee17928630b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:20.755398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.755280 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-istio-token" pod "655aebda-9cbb-4aae-b560-ee17928630b9" (UID: "655aebda-9cbb-4aae-b560-ee17928630b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:20.755828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.755796 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-envoy" pod "655aebda-9cbb-4aae-b560-ee17928630b9" (UID: "655aebda-9cbb-4aae-b560-ee17928630b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.843052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.843018 5001 generic.go:356] "Generic (PLEG): container finished" podID="79f0d6ea-d80a-4c14-8135-9647816f7eff" containerID="899a50b95f0e71d909e1f8cf463c3045883f4231f2a562a2e893b2dfde745c91" exitCode=0 Oct 30 06:23:20.843224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.843097 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" event={"ID":"79f0d6ea-d80a-4c14-8135-9647816f7eff","Type":"ContainerDied","Data":"899a50b95f0e71d909e1f8cf463c3045883f4231f2a562a2e893b2dfde745c91"} Oct 30 06:23:20.852268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.849086 5001 generic.go:356] "Generic (PLEG): container finished" podID="655aebda-9cbb-4aae-b560-ee17928630b9" containerID="7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8" exitCode=0 Oct 30 06:23:20.852268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.849249 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" event={"ID":"655aebda-9cbb-4aae-b560-ee17928630b9","Type":"ContainerDied","Data":"7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8"} Oct 30 06:23:20.852268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.849359 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" event={"ID":"655aebda-9cbb-4aae-b560-ee17928630b9","Type":"ContainerDied","Data":"355a81bcc1c0eaa45a5dad4e9abbc7be8b3171852b9fc02492ba9d4c50bb05fa"} Oct 30 06:23:20.852268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.849401 5001 scope.go:122] "RemoveContainer" containerID="7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8" Oct 30 06:23:20.852268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.849602 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn" Oct 30 06:23:20.853194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.853166 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l6nk\" (UniqueName: \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-kube-api-access-8l6nk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.853329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.853198 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.853329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.853218 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655aebda-9cbb-4aae-b560-ee17928630b9-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.853329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.853230 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.853329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.853241 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655aebda-9cbb-4aae-b560-ee17928630b9-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.853329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.853252 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655aebda-9cbb-4aae-b560-ee17928630b9-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.871477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.871449 5001 scope.go:122] "RemoveContainer" containerID="7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8" Oct 30 06:23:20.871849 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:23:20.871818 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8\": container with ID starting with 7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8 not found: ID does not exist" containerID="7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8" Oct 30 06:23:20.871978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.871860 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8"} err="failed to get container status \"7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8\": rpc error: code = NotFound desc = could not find container \"7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8\": container with ID starting with 7244143d8d5c98330eb969066911c2f8d3fb5d9e5c688e1de50c1a8c3a0cb5d8 not found: ID does not exist" Oct 30 06:23:20.922858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.922826 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn"] Oct 30 06:23:20.930180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:20.930153 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-bbd0e5-openshift-default-7d48679bbf-kdltn"] Oct 30 06:23:21.940494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:21.940434 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2"] Oct 30 06:23:21.941120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:21.941083 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" containerID="cri-o://185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356" gracePeriod=30 Oct 30 06:23:22.220327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.216801 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655aebda-9cbb-4aae-b560-ee17928630b9" path="/var/lib/kubelet/pods/655aebda-9cbb-4aae-b560-ee17928630b9/volumes" Oct 30 06:23:22.242135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.242107 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:22.312989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.312946 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-openshift-ca\") pod \"79f0d6ea-d80a-4c14-8135-9647816f7eff\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " Oct 30 06:23:22.313192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.313033 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-xks-ca\") pod \"79f0d6ea-d80a-4c14-8135-9647816f7eff\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " Oct 30 06:23:22.313192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.313139 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f0d6ea-d80a-4c14-8135-9647816f7eff-kube-api-access-sk755\" (UniqueName: \"kubernetes.io/projected/79f0d6ea-d80a-4c14-8135-9647816f7eff-kube-api-access-sk755\") pod \"79f0d6ea-d80a-4c14-8135-9647816f7eff\" (UID: \"79f0d6ea-d80a-4c14-8135-9647816f7eff\") " Oct 30 06:23:22.313482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.313454 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-openshift-ca" pod "79f0d6ea-d80a-4c14-8135-9647816f7eff" (UID: "79f0d6ea-d80a-4c14-8135-9647816f7eff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:22.313861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.313838 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:22.315838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.315810 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-xks-ca" pod "79f0d6ea-d80a-4c14-8135-9647816f7eff" (UID: "79f0d6ea-d80a-4c14-8135-9647816f7eff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:22.316201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.316179 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f0d6ea-d80a-4c14-8135-9647816f7eff-kube-api-access-sk755" pod "79f0d6ea-d80a-4c14-8135-9647816f7eff" (UID: "79f0d6ea-d80a-4c14-8135-9647816f7eff"). InnerVolumeSpecName "kube-api-access-sk755". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:22.415068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.414986 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f0d6ea-d80a-4c14-8135-9647816f7eff-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:22.415068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.415019 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk755\" (UniqueName: \"kubernetes.io/projected/79f0d6ea-d80a-4c14-8135-9647816f7eff-kube-api-access-sk755\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:22.483908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.483871 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx"] Oct 30 06:23:22.484723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.484698 5001 cpu_manager.go:414] "containerMap: removing container" podUID="79f0d6ea-d80a-4c14-8135-9647816f7eff" containerName="revoke-keys" Oct 30 06:23:22.484723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.484723 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f0d6ea-d80a-4c14-8135-9647816f7eff" containerName="revoke-keys" Oct 30 06:23:22.484936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.484760 5001 cpu_manager.go:414] "containerMap: removing container" podUID="655aebda-9cbb-4aae-b560-ee17928630b9" containerName="istio-proxy" Oct 30 06:23:22.484936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.484768 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655aebda-9cbb-4aae-b560-ee17928630b9" containerName="istio-proxy" Oct 30 06:23:22.485047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.484949 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f0d6ea-d80a-4c14-8135-9647816f7eff" containerName="revoke-keys" Oct 30 06:23:22.485047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.484967 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="655aebda-9cbb-4aae-b560-ee17928630b9" containerName="istio-proxy" Oct 30 06:23:22.488987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.488960 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:22.507164 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.507020 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx"] Oct 30 06:23:22.618828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.618781 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:22.619019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.618842 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:22.619019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.618962 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvxmt\" (UniqueName: \"kubernetes.io/projected/4d860fce-4aaa-4719-9adb-e5c7b40892bc-kube-api-access-rvxmt\") pod \"maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:22.720294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.720201 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:22.720294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.720252 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:22.720568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.720356 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvxmt\" (UniqueName: \"kubernetes.io/projected/4d860fce-4aaa-4719-9adb-e5c7b40892bc-kube-api-access-rvxmt\") pod \"maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:22.721214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.721037 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:22.722971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.722947 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:22.743802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.743771 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvxmt\" (UniqueName: \"kubernetes.io/projected/4d860fce-4aaa-4719-9adb-e5c7b40892bc-kube-api-access-rvxmt\") pod \"maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:22.806209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.806169 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:22.881610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.881500 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" event={"ID":"79f0d6ea-d80a-4c14-8135-9647816f7eff","Type":"ContainerDied","Data":"f51cd2ccba28322d6a40066ec0b5cfb490ad817ddaea296c3faba4b3021e4aca"} Oct 30 06:23:22.881610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.881572 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f51cd2ccba28322d6a40066ec0b5cfb490ad817ddaea296c3faba4b3021e4aca" Oct 30 06:23:22.887994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:22.884018 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24" Oct 30 06:23:23.140324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:23.140279 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx"] Oct 30 06:23:23.143833 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:23:23.143804 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d860fce_4aaa_4719_9adb_e5c7b40892bc.slice/crio-e0e3ff7198cb63d373f876e37cf97ef74aaf988dca7dee035e6e94edf61434f7 WatchSource:0}: Error finding container e0e3ff7198cb63d373f876e37cf97ef74aaf988dca7dee035e6e94edf61434f7: Status 404 returned error can't find the container with id e0e3ff7198cb63d373f876e37cf97ef74aaf988dca7dee035e6e94edf61434f7 Oct 30 06:23:23.909041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:23.908999 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" event={"ID":"4d860fce-4aaa-4719-9adb-e5c7b40892bc","Type":"ContainerStarted","Data":"2846c490b1bacd7d33b4efb866cdfc0674e2fb44477773e407fb8b77df6e1541"} Oct 30 06:23:23.909254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:23.909100 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" event={"ID":"4d860fce-4aaa-4719-9adb-e5c7b40892bc","Type":"ContainerStarted","Data":"e0e3ff7198cb63d373f876e37cf97ef74aaf988dca7dee035e6e94edf61434f7"} Oct 30 06:23:23.949168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:23.949110 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" podStartSLOduration=1.9490951060000001 podStartE2EDuration="1.949095106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:23.948546452 +0000 UTC m=+1817.944503461" watchObservedRunningTime="2026-10-30 06:23:23.949095106 +0000 UTC m=+1817.945052117" Oct 30 06:23:24.919133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:24.919102 5001 generic.go:356] "Generic (PLEG): container finished" podID="446c1aec-14d5-468e-887d-39c6579f389c" containerID="51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343" exitCode=0 Oct 30 06:23:24.919737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:24.919176 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" event={"ID":"446c1aec-14d5-468e-887d-39c6579f389c","Type":"ContainerDied","Data":"51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343"} Oct 30 06:23:24.919737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:24.919705 5001 scope.go:122] "RemoveContainer" containerID="51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343" Oct 30 06:23:24.921071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:24.921049 5001 generic.go:356] "Generic (PLEG): container finished" podID="4d860fce-4aaa-4719-9adb-e5c7b40892bc" containerID="2846c490b1bacd7d33b4efb866cdfc0674e2fb44477773e407fb8b77df6e1541" exitCode=0 Oct 30 06:23:24.921199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:24.921145 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" event={"ID":"4d860fce-4aaa-4719-9adb-e5c7b40892bc","Type":"ContainerDied","Data":"2846c490b1bacd7d33b4efb866cdfc0674e2fb44477773e407fb8b77df6e1541"} Oct 30 06:23:24.921513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:24.921482 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:23:24.921608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:24.921576 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:23:24.921670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:24.921628 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:23:25.931560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:25.931523 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" event={"ID":"446c1aec-14d5-468e-887d-39c6579f389c","Type":"ContainerStarted","Data":"b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3"} Oct 30 06:23:26.140761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.140738 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:26.283897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.283777 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-xks-ca\") pod \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " Oct 30 06:23:26.284114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.283936 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-openshift-ca\") pod \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " Oct 30 06:23:26.284114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.284058 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d860fce-4aaa-4719-9adb-e5c7b40892bc-kube-api-access-rvxmt\" (UniqueName: \"kubernetes.io/projected/4d860fce-4aaa-4719-9adb-e5c7b40892bc-kube-api-access-rvxmt\") pod \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\" (UID: \"4d860fce-4aaa-4719-9adb-e5c7b40892bc\") " Oct 30 06:23:26.290942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.288784 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-openshift-ca" pod "4d860fce-4aaa-4719-9adb-e5c7b40892bc" (UID: "4d860fce-4aaa-4719-9adb-e5c7b40892bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:26.292406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.291350 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-xks-ca" pod "4d860fce-4aaa-4719-9adb-e5c7b40892bc" (UID: "4d860fce-4aaa-4719-9adb-e5c7b40892bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:26.295527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.295499 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d860fce-4aaa-4719-9adb-e5c7b40892bc-kube-api-access-rvxmt" pod "4d860fce-4aaa-4719-9adb-e5c7b40892bc" (UID: "4d860fce-4aaa-4719-9adb-e5c7b40892bc"). InnerVolumeSpecName "kube-api-access-rvxmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:26.386501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.386464 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvxmt\" (UniqueName: \"kubernetes.io/projected/4d860fce-4aaa-4719-9adb-e5c7b40892bc-kube-api-access-rvxmt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:26.386501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.386498 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:26.386751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.386511 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d860fce-4aaa-4719-9adb-e5c7b40892bc-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:26.428335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.428268 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:23:26.428508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.428493 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 06:23:26.428561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.428537 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 06:23:26.941079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.941049 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" Oct 30 06:23:26.941523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.941089 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx" event={"ID":"4d860fce-4aaa-4719-9adb-e5c7b40892bc","Type":"ContainerDied","Data":"e0e3ff7198cb63d373f876e37cf97ef74aaf988dca7dee035e6e94edf61434f7"} Oct 30 06:23:26.941523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:26.941138 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0e3ff7198cb63d373f876e37cf97ef74aaf988dca7dee035e6e94edf61434f7" Oct 30 06:23:27.303601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.303573 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:23:27.417629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.417597 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-data\") pod \"975f7621-69e1-4a2c-8f93-871c39b4ca75\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " Oct 30 06:23:27.417816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.417677 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-kube-api-access-ssd8j\" (UniqueName: \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-kube-api-access-ssd8j\") pod \"975f7621-69e1-4a2c-8f93-871c39b4ca75\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " Oct 30 06:23:27.417816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.417704 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-credential-socket\") pod \"975f7621-69e1-4a2c-8f93-871c39b4ca75\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " Oct 30 06:23:27.417816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.417769 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-podinfo\") pod \"975f7621-69e1-4a2c-8f93-871c39b4ca75\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " Oct 30 06:23:27.418008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.417820 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975f7621-69e1-4a2c-8f93-871c39b4ca75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975f7621-69e1-4a2c-8f93-871c39b4ca75-istiod-ca-cert\") pod \"975f7621-69e1-4a2c-8f93-871c39b4ca75\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " Oct 30 06:23:27.418008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.417866 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-token\" (UniqueName: \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-token\") pod \"975f7621-69e1-4a2c-8f93-871c39b4ca75\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " Oct 30 06:23:27.418008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.417893 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-envoy\") pod \"975f7621-69e1-4a2c-8f93-871c39b4ca75\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " Oct 30 06:23:27.418008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.417931 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-socket\") pod \"975f7621-69e1-4a2c-8f93-871c39b4ca75\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " Oct 30 06:23:27.418008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.417937 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-data" pod "975f7621-69e1-4a2c-8f93-871c39b4ca75" (UID: "975f7621-69e1-4a2c-8f93-871c39b4ca75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:27.418865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.418164 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-socket" pod "975f7621-69e1-4a2c-8f93-871c39b4ca75" (UID: "975f7621-69e1-4a2c-8f93-871c39b4ca75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:27.418865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.418223 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-certs\") pod \"975f7621-69e1-4a2c-8f93-871c39b4ca75\" (UID: \"975f7621-69e1-4a2c-8f93-871c39b4ca75\") " Oct 30 06:23:27.418865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.418535 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-credential-socket" pod "975f7621-69e1-4a2c-8f93-871c39b4ca75" (UID: "975f7621-69e1-4a2c-8f93-871c39b4ca75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:27.418865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.418624 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975f7621-69e1-4a2c-8f93-871c39b4ca75-istiod-ca-cert" pod "975f7621-69e1-4a2c-8f93-871c39b4ca75" (UID: "975f7621-69e1-4a2c-8f93-871c39b4ca75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:27.418865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.418694 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-certs" pod "975f7621-69e1-4a2c-8f93-871c39b4ca75" (UID: "975f7621-69e1-4a2c-8f93-871c39b4ca75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:27.419141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.419116 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.419220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.419142 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.419220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.419157 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.419220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.419171 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.419220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.419187 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975f7621-69e1-4a2c-8f93-871c39b4ca75-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.420905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.420883 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-token" pod "975f7621-69e1-4a2c-8f93-871c39b4ca75" (UID: "975f7621-69e1-4a2c-8f93-871c39b4ca75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:27.421002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.420912 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-kube-api-access-ssd8j" pod "975f7621-69e1-4a2c-8f93-871c39b4ca75" (UID: "975f7621-69e1-4a2c-8f93-871c39b4ca75"). InnerVolumeSpecName "kube-api-access-ssd8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:27.421063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.421047 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-envoy" pod "975f7621-69e1-4a2c-8f93-871c39b4ca75" (UID: "975f7621-69e1-4a2c-8f93-871c39b4ca75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:27.421416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.421395 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-podinfo" pod "975f7621-69e1-4a2c-8f93-871c39b4ca75" (UID: "975f7621-69e1-4a2c-8f93-871c39b4ca75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:27.428854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.428824 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 06:23:27.428955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.428873 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 06:23:27.520975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.520871 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssd8j\" (UniqueName: \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-kube-api-access-ssd8j\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.520975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.520906 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.520975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.520920 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.520975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.520931 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975f7621-69e1-4a2c-8f93-871c39b4ca75-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.994331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.994096 5001 generic.go:356] "Generic (PLEG): container finished" podID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerID="185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356" exitCode=0 Oct 30 06:23:27.994850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.994372 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" event={"ID":"975f7621-69e1-4a2c-8f93-871c39b4ca75","Type":"ContainerDied","Data":"185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356"} Oct 30 06:23:27.994850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.994431 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" event={"ID":"975f7621-69e1-4a2c-8f93-871c39b4ca75","Type":"ContainerDied","Data":"5a306016b6fab41406897b94e968f7b5908d88d647b54535e86cd8a951a432b3"} Oct 30 06:23:27.994850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.994463 5001 scope.go:122] "RemoveContainer" containerID="185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356" Oct 30 06:23:27.994850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:27.994632 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2" Oct 30 06:23:28.024328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:28.023879 5001 scope.go:122] "RemoveContainer" containerID="bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b" Oct 30 06:23:28.039726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:28.039706 5001 scope.go:122] "RemoveContainer" containerID="185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356" Oct 30 06:23:28.039926 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:23:28.039900 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356\": container with ID starting with 185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356 not found: ID does not exist" containerID="185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356" Oct 30 06:23:28.039983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:28.039934 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356"} err="failed to get container status \"185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356\": rpc error: code = NotFound desc = could not find container \"185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356\": container with ID starting with 185524edbd377146aa383ff66591624b2498b63603e619202f1990e2d1cb5356 not found: ID does not exist" Oct 30 06:23:28.039983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:28.039951 5001 scope.go:122] "RemoveContainer" containerID="bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b" Oct 30 06:23:28.040156 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:23:28.040135 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b\": container with ID starting with bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b not found: ID does not exist" containerID="bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b" Oct 30 06:23:28.040207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:28.040158 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b"} err="failed to get container status \"bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b\": rpc error: code = NotFound desc = could not find container \"bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b\": container with ID starting with bfb36946320dad7dc954a92aff43c5acff3721fd1231ffff58f8516d0f27027b not found: ID does not exist" Oct 30 06:23:28.086853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:28.086816 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2"] Oct 30 06:23:28.095652 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:28.095623 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-520bc4-openshift-default-77fb485c89-nqtp2"] Oct 30 06:23:28.127295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:28.127255 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" path="/var/lib/kubelet/pods/975f7621-69e1-4a2c-8f93-871c39b4ca75/volumes" Oct 30 06:23:28.429523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:28.429487 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 06:23:28.429723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:28.429550 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 06:23:29.086176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:29.086130 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj"] Oct 30 06:23:29.429391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:29.428752 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 06:23:29.429391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:29.428835 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 06:23:30.429376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:30.429334 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 06:23:30.429826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:30.429393 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 06:23:31.428533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:31.428492 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 06:23:31.428742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:31.428563 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 06:23:32.454145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:32.452581 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:23:32.454145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:32.453931 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:23:32.455013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:32.454988 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:23:32.481462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:32.481425 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2"] Oct 30 06:23:32.481974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:32.481930 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" podUID="d923d506-2ebd-470d-b04d-ffa47379c56a" containerName="payload-processing" containerID="cri-o://6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007" gracePeriod=30 Oct 30 06:23:32.604634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:32.602493 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5"] Oct 30 06:23:32.604634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:32.602898 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" podUID="1ddedc39-8fc0-4569-b483-6af92b36eacb" containerName="payload-pre-processing" containerID="cri-o://324ae686b42bdc043a74cbcdb3ade810c1c308c04ca351e3e5c5cef2b8b56f8b" gracePeriod=30 Oct 30 06:23:33.027551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.022126 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr"] Oct 30 06:23:33.055368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.055340 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:23:33.064350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.064293 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb"] Oct 30 06:23:33.093463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.093256 5001 generic.go:356] "Generic (PLEG): container finished" podID="1ddedc39-8fc0-4569-b483-6af92b36eacb" containerID="324ae686b42bdc043a74cbcdb3ade810c1c308c04ca351e3e5c5cef2b8b56f8b" exitCode=0 Oct 30 06:23:33.093947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.093548 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" event={"ID":"1ddedc39-8fc0-4569-b483-6af92b36eacb","Type":"ContainerDied","Data":"324ae686b42bdc043a74cbcdb3ade810c1c308c04ca351e3e5c5cef2b8b56f8b"} Oct 30 06:23:33.095358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.095290 5001 generic.go:356] "Generic (PLEG): container finished" podID="d923d506-2ebd-470d-b04d-ffa47379c56a" containerID="6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007" exitCode=0 Oct 30 06:23:33.096749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.096380 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" Oct 30 06:23:33.096749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.096558 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" event={"ID":"d923d506-2ebd-470d-b04d-ffa47379c56a","Type":"ContainerDied","Data":"6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007"} Oct 30 06:23:33.096749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.096598 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2" event={"ID":"d923d506-2ebd-470d-b04d-ffa47379c56a","Type":"ContainerDied","Data":"9e3219607f3e0ec091b0e935b67f91960dd242ad38cb94412732fa2a4cf967c2"} Oct 30 06:23:33.096749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.096625 5001 scope.go:122] "RemoveContainer" containerID="6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007" Oct 30 06:23:33.112586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.112487 5001 scope.go:122] "RemoveContainer" containerID="6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007" Oct 30 06:23:33.113150 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:23:33.113120 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007\": container with ID starting with 6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007 not found: ID does not exist" containerID="6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007" Oct 30 06:23:33.113821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.113289 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007"} err="failed to get container status \"6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007\": rpc error: code = NotFound desc = could not find container \"6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007\": container with ID starting with 6e7643b14145fda54bf96e5073d3a3ffbda3fcd2713ca701be3c5b5ef47b8007 not found: ID does not exist" Oct 30 06:23:33.137591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.137556 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d923d506-2ebd-470d-b04d-ffa47379c56a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d923d506-2ebd-470d-b04d-ffa47379c56a-plugins-config-volume\") pod \"d923d506-2ebd-470d-b04d-ffa47379c56a\" (UID: \"d923d506-2ebd-470d-b04d-ffa47379c56a\") " Oct 30 06:23:33.137920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.137664 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d923d506-2ebd-470d-b04d-ffa47379c56a-kube-api-access-4csp7\" (UniqueName: \"kubernetes.io/projected/d923d506-2ebd-470d-b04d-ffa47379c56a-kube-api-access-4csp7\") pod \"d923d506-2ebd-470d-b04d-ffa47379c56a\" (UID: \"d923d506-2ebd-470d-b04d-ffa47379c56a\") " Oct 30 06:23:33.138543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.138515 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d923d506-2ebd-470d-b04d-ffa47379c56a-plugins-config-volume" pod "d923d506-2ebd-470d-b04d-ffa47379c56a" (UID: "d923d506-2ebd-470d-b04d-ffa47379c56a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:33.141625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.141599 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d923d506-2ebd-470d-b04d-ffa47379c56a-kube-api-access-4csp7" pod "d923d506-2ebd-470d-b04d-ffa47379c56a" (UID: "d923d506-2ebd-470d-b04d-ffa47379c56a"). InnerVolumeSpecName "kube-api-access-4csp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:33.141981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.141955 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d923d506-2ebd-470d-b04d-ffa47379c56a-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:33.142128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.141988 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4csp7\" (UniqueName: \"kubernetes.io/projected/d923d506-2ebd-470d-b04d-ffa47379c56a-kube-api-access-4csp7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:33.174433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.174395 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l"] Oct 30 06:23:33.180588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.180557 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-532f3b-24v4l"] Oct 30 06:23:33.205074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.205046 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:23:33.243151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.243118 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ddedc39-8fc0-4569-b483-6af92b36eacb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ddedc39-8fc0-4569-b483-6af92b36eacb-plugins-config-volume\") pod \"1ddedc39-8fc0-4569-b483-6af92b36eacb\" (UID: \"1ddedc39-8fc0-4569-b483-6af92b36eacb\") " Oct 30 06:23:33.270639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.270553 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddedc39-8fc0-4569-b483-6af92b36eacb-kube-api-access-9gcsn\" (UniqueName: \"kubernetes.io/projected/1ddedc39-8fc0-4569-b483-6af92b36eacb-kube-api-access-9gcsn\") pod \"1ddedc39-8fc0-4569-b483-6af92b36eacb\" (UID: \"1ddedc39-8fc0-4569-b483-6af92b36eacb\") " Oct 30 06:23:33.272262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.272229 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ddedc39-8fc0-4569-b483-6af92b36eacb-plugins-config-volume" pod "1ddedc39-8fc0-4569-b483-6af92b36eacb" (UID: "1ddedc39-8fc0-4569-b483-6af92b36eacb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:33.294547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.294507 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddedc39-8fc0-4569-b483-6af92b36eacb-kube-api-access-9gcsn" pod "1ddedc39-8fc0-4569-b483-6af92b36eacb" (UID: "1ddedc39-8fc0-4569-b483-6af92b36eacb"). InnerVolumeSpecName "kube-api-access-9gcsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:33.372992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.372957 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ddedc39-8fc0-4569-b483-6af92b36eacb-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:33.372992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.372992 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gcsn\" (UniqueName: \"kubernetes.io/projected/1ddedc39-8fc0-4569-b483-6af92b36eacb-kube-api-access-9gcsn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:33.444996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.444963 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2"] Oct 30 06:23:33.450829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:33.450801 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-532f3b-65596d649b-6cng2"] Oct 30 06:23:34.086760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.086715 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" podUID="c8821eff-dc4e-456a-9112-5fee294c08ac" containerName="payload-processing" containerID="cri-o://72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f" gracePeriod=25 Oct 30 06:23:34.107119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.107093 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" Oct 30 06:23:34.107285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.107083 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5" event={"ID":"1ddedc39-8fc0-4569-b483-6af92b36eacb","Type":"ContainerDied","Data":"377c33ffb00a864b38c1f23c6e47e73a1256c23a06079ad1aa3b75fc07dd0092"} Oct 30 06:23:34.107285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.107238 5001 scope.go:122] "RemoveContainer" containerID="324ae686b42bdc043a74cbcdb3ade810c1c308c04ca351e3e5c5cef2b8b56f8b" Oct 30 06:23:34.117872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.117849 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935f06ff-5347-4957-8d08-748c9d606a03" path="/var/lib/kubelet/pods/935f06ff-5347-4957-8d08-748c9d606a03/volumes" Oct 30 06:23:34.118250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.118237 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d923d506-2ebd-470d-b04d-ffa47379c56a" path="/var/lib/kubelet/pods/d923d506-2ebd-470d-b04d-ffa47379c56a/volumes" Oct 30 06:23:34.216062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.216033 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5"] Oct 30 06:23:34.226854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.226818 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-532f3b-7b46c6c667-hlgh5"] Oct 30 06:23:34.378115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.378089 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:23:34.493666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.493623 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8821eff-dc4e-456a-9112-5fee294c08ac-kube-api-access-b95r2\" (UniqueName: \"kubernetes.io/projected/c8821eff-dc4e-456a-9112-5fee294c08ac-kube-api-access-b95r2\") pod \"c8821eff-dc4e-456a-9112-5fee294c08ac\" (UID: \"c8821eff-dc4e-456a-9112-5fee294c08ac\") " Oct 30 06:23:34.493872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.493724 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8821eff-dc4e-456a-9112-5fee294c08ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8821eff-dc4e-456a-9112-5fee294c08ac-plugins-config-volume\") pod \"c8821eff-dc4e-456a-9112-5fee294c08ac\" (UID: \"c8821eff-dc4e-456a-9112-5fee294c08ac\") " Oct 30 06:23:34.494216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.494191 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8821eff-dc4e-456a-9112-5fee294c08ac-plugins-config-volume" pod "c8821eff-dc4e-456a-9112-5fee294c08ac" (UID: "c8821eff-dc4e-456a-9112-5fee294c08ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:34.496202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.496173 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8821eff-dc4e-456a-9112-5fee294c08ac-kube-api-access-b95r2" pod "c8821eff-dc4e-456a-9112-5fee294c08ac" (UID: "c8821eff-dc4e-456a-9112-5fee294c08ac"). InnerVolumeSpecName "kube-api-access-b95r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:34.595100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.595066 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b95r2\" (UniqueName: \"kubernetes.io/projected/c8821eff-dc4e-456a-9112-5fee294c08ac-kube-api-access-b95r2\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:34.595100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:34.595097 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8821eff-dc4e-456a-9112-5fee294c08ac-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:35.118796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:35.118763 5001 generic.go:356] "Generic (PLEG): container finished" podID="c8821eff-dc4e-456a-9112-5fee294c08ac" containerID="72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f" exitCode=0 Oct 30 06:23:35.119276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:35.118802 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" event={"ID":"c8821eff-dc4e-456a-9112-5fee294c08ac","Type":"ContainerDied","Data":"72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f"} Oct 30 06:23:35.119276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:35.118812 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" Oct 30 06:23:35.119276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:35.118836 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj" event={"ID":"c8821eff-dc4e-456a-9112-5fee294c08ac","Type":"ContainerDied","Data":"f6f9154cc133d0e3df5890e944f714f8b605e22cd594a9e1a6ea9b374438cd7b"} Oct 30 06:23:35.119276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:35.118855 5001 scope.go:122] "RemoveContainer" containerID="72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f" Oct 30 06:23:35.153545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:35.138895 5001 scope.go:122] "RemoveContainer" containerID="72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f" Oct 30 06:23:35.153545 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:23:35.139393 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f\": container with ID starting with 72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f not found: ID does not exist" containerID="72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f" Oct 30 06:23:35.153545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:35.139433 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f"} err="failed to get container status \"72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f\": rpc error: code = NotFound desc = could not find container \"72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f\": container with ID starting with 72291bc3cbb54bdc557a47c220df99c8bc2ce92a25c5dde01298fb33759ecd5f not found: ID does not exist" Oct 30 06:23:35.204866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:35.204769 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj"] Oct 30 06:23:35.213416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:35.213385 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f6a022-597b4db76-hf2jj"] Oct 30 06:23:36.118910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:36.118873 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ddedc39-8fc0-4569-b483-6af92b36eacb" path="/var/lib/kubelet/pods/1ddedc39-8fc0-4569-b483-6af92b36eacb/volumes" Oct 30 06:23:36.119350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:36.119254 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8821eff-dc4e-456a-9112-5fee294c08ac" path="/var/lib/kubelet/pods/c8821eff-dc4e-456a-9112-5fee294c08ac/volumes" Oct 30 06:23:38.022697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.022655 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" podUID="bdd08ddf-057c-49ac-a89a-0250c6c3867f" containerName="payload-processing" containerID="cri-o://c1129d7cba57576c309762d481bb8d23e4a19f2ebb0b0e2ece28d3de07335d8e" gracePeriod=25 Oct 30 06:23:38.064768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.064734 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" podUID="ec461549-4c1d-4a0e-ab11-baf044529086" containerName="payload-pre-processing" containerID="cri-o://d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd" gracePeriod=25 Oct 30 06:23:38.144150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.144124 5001 generic.go:356] "Generic (PLEG): container finished" podID="bdd08ddf-057c-49ac-a89a-0250c6c3867f" containerID="c1129d7cba57576c309762d481bb8d23e4a19f2ebb0b0e2ece28d3de07335d8e" exitCode=0 Oct 30 06:23:38.144291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.144202 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" event={"ID":"bdd08ddf-057c-49ac-a89a-0250c6c3867f","Type":"ContainerDied","Data":"c1129d7cba57576c309762d481bb8d23e4a19f2ebb0b0e2ece28d3de07335d8e"} Oct 30 06:23:38.267427 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:23:38.267392 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:23:38.439136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.439111 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:23:38.443552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.443532 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:23:38.552263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.552233 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec461549-4c1d-4a0e-ab11-baf044529086-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec461549-4c1d-4a0e-ab11-baf044529086-plugins-config-volume\") pod \"ec461549-4c1d-4a0e-ab11-baf044529086\" (UID: \"ec461549-4c1d-4a0e-ab11-baf044529086\") " Oct 30 06:23:38.552466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.552293 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec461549-4c1d-4a0e-ab11-baf044529086-kube-api-access-8556h\" (UniqueName: \"kubernetes.io/projected/ec461549-4c1d-4a0e-ab11-baf044529086-kube-api-access-8556h\") pod \"ec461549-4c1d-4a0e-ab11-baf044529086\" (UID: \"ec461549-4c1d-4a0e-ab11-baf044529086\") " Oct 30 06:23:38.552466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.552404 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd08ddf-057c-49ac-a89a-0250c6c3867f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd08ddf-057c-49ac-a89a-0250c6c3867f-plugins-config-volume\") pod \"bdd08ddf-057c-49ac-a89a-0250c6c3867f\" (UID: \"bdd08ddf-057c-49ac-a89a-0250c6c3867f\") " Oct 30 06:23:38.552678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.552525 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd08ddf-057c-49ac-a89a-0250c6c3867f-kube-api-access-hsvct\" (UniqueName: \"kubernetes.io/projected/bdd08ddf-057c-49ac-a89a-0250c6c3867f-kube-api-access-hsvct\") pod \"bdd08ddf-057c-49ac-a89a-0250c6c3867f\" (UID: \"bdd08ddf-057c-49ac-a89a-0250c6c3867f\") " Oct 30 06:23:38.553207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.553174 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd08ddf-057c-49ac-a89a-0250c6c3867f-plugins-config-volume" pod "bdd08ddf-057c-49ac-a89a-0250c6c3867f" (UID: "bdd08ddf-057c-49ac-a89a-0250c6c3867f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:38.553921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.553891 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec461549-4c1d-4a0e-ab11-baf044529086-plugins-config-volume" pod "ec461549-4c1d-4a0e-ab11-baf044529086" (UID: "ec461549-4c1d-4a0e-ab11-baf044529086"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:38.555786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.555761 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd08ddf-057c-49ac-a89a-0250c6c3867f-kube-api-access-hsvct" pod "bdd08ddf-057c-49ac-a89a-0250c6c3867f" (UID: "bdd08ddf-057c-49ac-a89a-0250c6c3867f"). InnerVolumeSpecName "kube-api-access-hsvct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:38.556684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.556663 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec461549-4c1d-4a0e-ab11-baf044529086-kube-api-access-8556h" pod "ec461549-4c1d-4a0e-ab11-baf044529086" (UID: "ec461549-4c1d-4a0e-ab11-baf044529086"). InnerVolumeSpecName "kube-api-access-8556h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:38.654446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.654404 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec461549-4c1d-4a0e-ab11-baf044529086-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:38.654446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.654441 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8556h\" (UniqueName: \"kubernetes.io/projected/ec461549-4c1d-4a0e-ab11-baf044529086-kube-api-access-8556h\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:38.654446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.654456 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd08ddf-057c-49ac-a89a-0250c6c3867f-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:38.654792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:38.654470 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsvct\" (UniqueName: \"kubernetes.io/projected/bdd08ddf-057c-49ac-a89a-0250c6c3867f-kube-api-access-hsvct\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:39.153612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.153572 5001 generic.go:356] "Generic (PLEG): container finished" podID="ec461549-4c1d-4a0e-ab11-baf044529086" containerID="d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd" exitCode=0 Oct 30 06:23:39.153612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.153616 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" Oct 30 06:23:39.154142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.153654 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" event={"ID":"ec461549-4c1d-4a0e-ab11-baf044529086","Type":"ContainerDied","Data":"d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd"} Oct 30 06:23:39.154142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.153729 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb" event={"ID":"ec461549-4c1d-4a0e-ab11-baf044529086","Type":"ContainerDied","Data":"2edd0a32f80b46595c69288dcff149b2436cc0aac2456c2a7e2c5c96e60cfac4"} Oct 30 06:23:39.154142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.153760 5001 scope.go:122] "RemoveContainer" containerID="d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd" Oct 30 06:23:39.155849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.155810 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" event={"ID":"bdd08ddf-057c-49ac-a89a-0250c6c3867f","Type":"ContainerDied","Data":"2ea0d30ec3cd308a4f0359edc99da8f717a367dd20aa64a8c133f18bbc8e66f1"} Oct 30 06:23:39.155953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.155801 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr" Oct 30 06:23:39.170134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.170116 5001 scope.go:122] "RemoveContainer" containerID="d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd" Oct 30 06:23:39.170615 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:23:39.170591 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd\": container with ID starting with d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd not found: ID does not exist" containerID="d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd" Oct 30 06:23:39.170699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.170626 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd"} err="failed to get container status \"d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd\": rpc error: code = NotFound desc = could not find container \"d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd\": container with ID starting with d67ff808e4deb0a883d4a3d5abd799308e3ec312a6bdf5324a2bf8355d05a5cd not found: ID does not exist" Oct 30 06:23:39.170699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.170647 5001 scope.go:122] "RemoveContainer" containerID="c1129d7cba57576c309762d481bb8d23e4a19f2ebb0b0e2ece28d3de07335d8e" Oct 30 06:23:39.210338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.210295 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr"] Oct 30 06:23:39.225351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.224021 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4dd310-555f786c8-v5rcr"] Oct 30 06:23:39.258041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.258003 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb"] Oct 30 06:23:39.263769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:39.263739 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4dd310-c5d7c7555-ml7gb"] Oct 30 06:23:40.119952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:40.119903 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd08ddf-057c-49ac-a89a-0250c6c3867f" path="/var/lib/kubelet/pods/bdd08ddf-057c-49ac-a89a-0250c6c3867f/volumes" Oct 30 06:23:40.120402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:40.120385 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec461549-4c1d-4a0e-ab11-baf044529086" path="/var/lib/kubelet/pods/ec461549-4c1d-4a0e-ab11-baf044529086/volumes" Oct 30 06:23:41.326435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.326378 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24"] Oct 30 06:23:41.376130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.366000 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f6a022-j8l24"] Oct 30 06:23:41.879548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.879505 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976"] Oct 30 06:23:41.880169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880148 5001 cpu_manager.go:414] "containerMap: removing container" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880170 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880199 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d923d506-2ebd-470d-b04d-ffa47379c56a" containerName="payload-processing" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880207 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d923d506-2ebd-470d-b04d-ffa47379c56a" containerName="payload-processing" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880224 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4d860fce-4aaa-4719-9adb-e5c7b40892bc" containerName="revoke-keys" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880232 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d860fce-4aaa-4719-9adb-e5c7b40892bc" containerName="revoke-keys" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880248 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1ddedc39-8fc0-4569-b483-6af92b36eacb" containerName="payload-pre-processing" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880256 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddedc39-8fc0-4569-b483-6af92b36eacb" containerName="payload-pre-processing" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880273 5001 cpu_manager.go:414] "containerMap: removing container" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880280 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880289 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ec461549-4c1d-4a0e-ab11-baf044529086" containerName="payload-pre-processing" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880296 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec461549-4c1d-4a0e-ab11-baf044529086" containerName="payload-pre-processing" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880328 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c8821eff-dc4e-456a-9112-5fee294c08ac" containerName="payload-processing" Oct 30 06:23:41.880299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880335 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8821eff-dc4e-456a-9112-5fee294c08ac" containerName="payload-processing" Oct 30 06:23:41.880998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880382 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bdd08ddf-057c-49ac-a89a-0250c6c3867f" containerName="payload-processing" Oct 30 06:23:41.880998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880391 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd08ddf-057c-49ac-a89a-0250c6c3867f" containerName="payload-processing" Oct 30 06:23:41.880998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880568 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" Oct 30 06:23:41.880998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880585 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d923d506-2ebd-470d-b04d-ffa47379c56a" containerName="payload-processing" Oct 30 06:23:41.880998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880600 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d860fce-4aaa-4719-9adb-e5c7b40892bc" containerName="revoke-keys" Oct 30 06:23:41.880998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880612 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddedc39-8fc0-4569-b483-6af92b36eacb" containerName="payload-pre-processing" Oct 30 06:23:41.880998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880622 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="975f7621-69e1-4a2c-8f93-871c39b4ca75" containerName="istio-proxy" Oct 30 06:23:41.880998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880633 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec461549-4c1d-4a0e-ab11-baf044529086" containerName="payload-pre-processing" Oct 30 06:23:41.880998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880641 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8821eff-dc4e-456a-9112-5fee294c08ac" containerName="payload-processing" Oct 30 06:23:41.880998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.880658 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd08ddf-057c-49ac-a89a-0250c6c3867f" containerName="payload-processing" Oct 30 06:23:41.885963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.885943 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:41.900543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.900521 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:41.904645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.904624 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976"] Oct 30 06:23:41.910706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.910686 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:23:41.910832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.910705 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:23:41.910832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:41.910725 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:23:42.010863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.010833 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:42.011079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.010874 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8v9q\" (UniqueName: \"kubernetes.io/projected/ac5d5603-8fed-47e4-9589-a5caa4f978e7-kube-api-access-g8v9q\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:42.011079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.011014 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:42.111799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.111768 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:42.111986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.111816 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8v9q\" (UniqueName: \"kubernetes.io/projected/ac5d5603-8fed-47e4-9589-a5caa4f978e7-kube-api-access-g8v9q\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:42.111986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.111868 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:42.112742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.112717 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:42.114391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.114367 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:42.119364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.119337 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f0d6ea-d80a-4c14-8135-9647816f7eff" path="/var/lib/kubelet/pods/79f0d6ea-d80a-4c14-8135-9647816f7eff/volumes" Oct 30 06:23:42.134609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.134553 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8v9q\" (UniqueName: \"kubernetes.io/projected/ac5d5603-8fed-47e4-9589-a5caa4f978e7-kube-api-access-g8v9q\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:42.200024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.199994 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:42.521857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:42.521833 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976"] Oct 30 06:23:42.524410 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:23:42.524379 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5d5603_8fed_47e4_9589_a5caa4f978e7.slice/crio-0aa777309705c3d76afa3354dbe1721cde99c99b0fad8915586a4f77244bd927 WatchSource:0}: Error finding container 0aa777309705c3d76afa3354dbe1721cde99c99b0fad8915586a4f77244bd927: Status 404 returned error can't find the container with id 0aa777309705c3d76afa3354dbe1721cde99c99b0fad8915586a4f77244bd927 Oct 30 06:23:43.196170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:43.196064 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" event={"ID":"ac5d5603-8fed-47e4-9589-a5caa4f978e7","Type":"ContainerStarted","Data":"1b72ddff188f7147eada9bc42eb41183f3ebba0b0c8de935243a32820dd9d089"} Oct 30 06:23:43.196170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:43.196135 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" event={"ID":"ac5d5603-8fed-47e4-9589-a5caa4f978e7","Type":"ContainerStarted","Data":"0aa777309705c3d76afa3354dbe1721cde99c99b0fad8915586a4f77244bd927"} Oct 30 06:23:44.205537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:44.205490 5001 generic.go:356] "Generic (PLEG): container finished" podID="ac5d5603-8fed-47e4-9589-a5caa4f978e7" containerID="1b72ddff188f7147eada9bc42eb41183f3ebba0b0c8de935243a32820dd9d089" exitCode=0 Oct 30 06:23:44.206016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:44.205555 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" event={"ID":"ac5d5603-8fed-47e4-9589-a5caa4f978e7","Type":"ContainerDied","Data":"1b72ddff188f7147eada9bc42eb41183f3ebba0b0c8de935243a32820dd9d089"} Oct 30 06:23:45.446233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:45.446205 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:45.578922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:45.578838 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-openshift-ca\") pod \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " Oct 30 06:23:45.578922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:45.578897 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-xks-ca\") pod \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " Oct 30 06:23:45.579133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:45.579101 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5d5603-8fed-47e4-9589-a5caa4f978e7-kube-api-access-g8v9q\" (UniqueName: \"kubernetes.io/projected/ac5d5603-8fed-47e4-9589-a5caa4f978e7-kube-api-access-g8v9q\") pod \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\" (UID: \"ac5d5603-8fed-47e4-9589-a5caa4f978e7\") " Oct 30 06:23:45.579336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:45.579281 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-openshift-ca" pod "ac5d5603-8fed-47e4-9589-a5caa4f978e7" (UID: "ac5d5603-8fed-47e4-9589-a5caa4f978e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:45.579519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:45.579500 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:45.581242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:45.581215 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-xks-ca" pod "ac5d5603-8fed-47e4-9589-a5caa4f978e7" (UID: "ac5d5603-8fed-47e4-9589-a5caa4f978e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:45.581391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:45.581290 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5d5603-8fed-47e4-9589-a5caa4f978e7-kube-api-access-g8v9q" pod "ac5d5603-8fed-47e4-9589-a5caa4f978e7" (UID: "ac5d5603-8fed-47e4-9589-a5caa4f978e7"). InnerVolumeSpecName "kube-api-access-g8v9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:45.681255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:45.681216 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8v9q\" (UniqueName: \"kubernetes.io/projected/ac5d5603-8fed-47e4-9589-a5caa4f978e7-kube-api-access-g8v9q\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:45.681255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:45.681253 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5d5603-8fed-47e4-9589-a5caa4f978e7-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:46.225101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:46.225074 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" Oct 30 06:23:46.225272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:46.225063 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976" event={"ID":"ac5d5603-8fed-47e4-9589-a5caa4f978e7","Type":"ContainerDied","Data":"0aa777309705c3d76afa3354dbe1721cde99c99b0fad8915586a4f77244bd927"} Oct 30 06:23:46.225272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:46.225172 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aa777309705c3d76afa3354dbe1721cde99c99b0fad8915586a4f77244bd927" Oct 30 06:23:49.576517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:49.576422 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng"] Oct 30 06:23:49.577100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:49.577016 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" containerID="cri-o://a50db4a48de95522a3384399ca880d594f0e0fc31a303df4ec2ab70044afb63c" gracePeriod=30 Oct 30 06:23:52.348175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:52.348140 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw"] Oct 30 06:23:52.368266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:52.368228 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r"] Oct 30 06:23:53.775399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:53.760270 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:23:54.300140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.300102 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb"] Oct 30 06:23:54.301092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.301066 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ac5d5603-8fed-47e4-9589-a5caa4f978e7" containerName="revoke-keys" Oct 30 06:23:54.301280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.301245 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5d5603-8fed-47e4-9589-a5caa4f978e7" containerName="revoke-keys" Oct 30 06:23:54.301495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.301479 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5d5603-8fed-47e4-9589-a5caa4f978e7" containerName="revoke-keys" Oct 30 06:23:54.306521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.306501 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:54.324991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.324959 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb"] Oct 30 06:23:54.330500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.330363 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:23:54.330500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.330452 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:23:54.330700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.330509 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:23:54.330700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.330560 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:54.344181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.344148 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fb235e-78b9-4f68-8698-e62153540562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb9d77b-sb4mb\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:54.344381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.344196 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fb235e-78b9-4f68-8698-e62153540562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb9d77b-sb4mb\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:54.344381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.344252 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4gl\" (UniqueName: \"kubernetes.io/projected/81fb235e-78b9-4f68-8698-e62153540562-kube-api-access-6p4gl\") pod \"maas-api-revoke-sub-dcb9d77b-sb4mb\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:54.446274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.446239 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fb235e-78b9-4f68-8698-e62153540562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb9d77b-sb4mb\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:54.446439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.446338 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fb235e-78b9-4f68-8698-e62153540562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb9d77b-sb4mb\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:54.446500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.446442 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4gl\" (UniqueName: \"kubernetes.io/projected/81fb235e-78b9-4f68-8698-e62153540562-kube-api-access-6p4gl\") pod \"maas-api-revoke-sub-dcb9d77b-sb4mb\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:54.446981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.446952 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fb235e-78b9-4f68-8698-e62153540562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb9d77b-sb4mb\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:54.448907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.448886 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fb235e-78b9-4f68-8698-e62153540562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb9d77b-sb4mb\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:54.469079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.469045 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4gl\" (UniqueName: \"kubernetes.io/projected/81fb235e-78b9-4f68-8698-e62153540562-kube-api-access-6p4gl\") pod \"maas-api-revoke-sub-dcb9d77b-sb4mb\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:54.628331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:54.623758 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:55.182049 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:23:55.178340 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fb235e_78b9_4f68_8698_e62153540562.slice/crio-670bb2799bd59711ad3bd38b3f6954d3d674b799b2c05e6c29c82d7354533874 WatchSource:0}: Error finding container 670bb2799bd59711ad3bd38b3f6954d3d674b799b2c05e6c29c82d7354533874: Status 404 returned error can't find the container with id 670bb2799bd59711ad3bd38b3f6954d3d674b799b2c05e6c29c82d7354533874 Oct 30 06:23:55.199885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.199842 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb"] Oct 30 06:23:55.330560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.330472 5001 generic.go:356] "Generic (PLEG): container finished" podID="448e441c-2924-4232-93c1-721a7b633880" containerID="a50db4a48de95522a3384399ca880d594f0e0fc31a303df4ec2ab70044afb63c" exitCode=0 Oct 30 06:23:55.330916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.330892 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" event={"ID":"448e441c-2924-4232-93c1-721a7b633880","Type":"ContainerDied","Data":"a50db4a48de95522a3384399ca880d594f0e0fc31a303df4ec2ab70044afb63c"} Oct 30 06:23:55.331095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.331082 5001 scope.go:122] "RemoveContainer" containerID="56caec93d7705a6453a81c29e86f5f34c576b1d66eb32d979b7373c9088b8209" Oct 30 06:23:55.341273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.341205 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" event={"ID":"81fb235e-78b9-4f68-8698-e62153540562","Type":"ContainerStarted","Data":"670bb2799bd59711ad3bd38b3f6954d3d674b799b2c05e6c29c82d7354533874"} Oct 30 06:23:55.486752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.486727 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:23:55.546496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.546468 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-envoy\") pod \"448e441c-2924-4232-93c1-721a7b633880\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " Oct 30 06:23:55.546721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.546601 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-istio-token\" (UniqueName: \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-istio-token\") pod \"448e441c-2924-4232-93c1-721a7b633880\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " Oct 30 06:23:55.546721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.546654 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-data\") pod \"448e441c-2924-4232-93c1-721a7b633880\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " Oct 30 06:23:55.546721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.546686 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-kube-api-access-7n6mx\" (UniqueName: \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-kube-api-access-7n6mx\") pod \"448e441c-2924-4232-93c1-721a7b633880\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " Oct 30 06:23:55.546900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.546760 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/448e441c-2924-4232-93c1-721a7b633880-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448e441c-2924-4232-93c1-721a7b633880-istio-podinfo\") pod \"448e441c-2924-4232-93c1-721a7b633880\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " Oct 30 06:23:55.546900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.546856 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-credential-socket\") pod \"448e441c-2924-4232-93c1-721a7b633880\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " Oct 30 06:23:55.546900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.546893 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-socket\") pod \"448e441c-2924-4232-93c1-721a7b633880\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " Oct 30 06:23:55.547068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.546968 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-certs\") pod \"448e441c-2924-4232-93c1-721a7b633880\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " Oct 30 06:23:55.547068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.547034 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448e441c-2924-4232-93c1-721a7b633880-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448e441c-2924-4232-93c1-721a7b633880-istiod-ca-cert\") pod \"448e441c-2924-4232-93c1-721a7b633880\" (UID: \"448e441c-2924-4232-93c1-721a7b633880\") " Oct 30 06:23:55.547408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.547198 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-data" pod "448e441c-2924-4232-93c1-721a7b633880" (UID: "448e441c-2924-4232-93c1-721a7b633880"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.547546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.547515 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448e441c-2924-4232-93c1-721a7b633880-istiod-ca-cert" pod "448e441c-2924-4232-93c1-721a7b633880" (UID: "448e441c-2924-4232-93c1-721a7b633880"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:55.547759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.547738 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448e441c-2924-4232-93c1-721a7b633880-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.547842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.547766 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.547842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.547787 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-credential-socket" pod "448e441c-2924-4232-93c1-721a7b633880" (UID: "448e441c-2924-4232-93c1-721a7b633880"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.547999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.547973 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-socket" pod "448e441c-2924-4232-93c1-721a7b633880" (UID: "448e441c-2924-4232-93c1-721a7b633880"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.548212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.548192 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-certs" pod "448e441c-2924-4232-93c1-721a7b633880" (UID: "448e441c-2924-4232-93c1-721a7b633880"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.549228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.549203 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/448e441c-2924-4232-93c1-721a7b633880-istio-podinfo" pod "448e441c-2924-4232-93c1-721a7b633880" (UID: "448e441c-2924-4232-93c1-721a7b633880"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:55.549589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.549561 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-kube-api-access-7n6mx" pod "448e441c-2924-4232-93c1-721a7b633880" (UID: "448e441c-2924-4232-93c1-721a7b633880"). InnerVolumeSpecName "kube-api-access-7n6mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:55.549806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.549783 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-envoy" pod "448e441c-2924-4232-93c1-721a7b633880" (UID: "448e441c-2924-4232-93c1-721a7b633880"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.550938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.550917 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-istio-token" pod "448e441c-2924-4232-93c1-721a7b633880" (UID: "448e441c-2924-4232-93c1-721a7b633880"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:55.619020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.618985 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx"] Oct 30 06:23:55.622520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.622491 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4dd310-j5zpx"] Oct 30 06:23:55.651359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.649773 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n6mx\" (UniqueName: \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-kube-api-access-7n6mx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.651359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.649888 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448e441c-2924-4232-93c1-721a7b633880-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.651359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.649912 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.651359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.650036 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.651359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.650058 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.651359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.650080 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448e441c-2924-4232-93c1-721a7b633880-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.651359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:55.650098 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448e441c-2924-4232-93c1-721a7b633880-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:56.119959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:56.119923 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d860fce-4aaa-4719-9adb-e5c7b40892bc" path="/var/lib/kubelet/pods/4d860fce-4aaa-4719-9adb-e5c7b40892bc/volumes" Oct 30 06:23:56.350972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:56.350936 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" Oct 30 06:23:56.351503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:56.350970 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng" event={"ID":"448e441c-2924-4232-93c1-721a7b633880","Type":"ContainerDied","Data":"7962a7d43910a391c7cfecd84ca10eadd6d2ba1a833fc1926fa844b128c20bcf"} Oct 30 06:23:56.351503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:56.351028 5001 scope.go:122] "RemoveContainer" containerID="a50db4a48de95522a3384399ca880d594f0e0fc31a303df4ec2ab70044afb63c" Oct 30 06:23:56.353253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:56.353230 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" event={"ID":"81fb235e-78b9-4f68-8698-e62153540562","Type":"ContainerStarted","Data":"f8a8dc257d324a47bb2209c41250ffc8fdc0fe9189b0923eb654ac39003ea39b"} Oct 30 06:23:56.456053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:56.456026 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng"] Oct 30 06:23:56.462831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:56.462804 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f6a022-openshift-default-6bb89969c-mflng"] Oct 30 06:23:56.504903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:56.504860 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" podStartSLOduration=2.50484521 podStartE2EDuration="2.50484521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:56.503063127 +0000 UTC m=+1850.499020135" watchObservedRunningTime="2026-10-30 06:23:56.50484521 +0000 UTC m=+1850.500802220" Oct 30 06:23:57.348938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.348900 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" podUID="42deeec4-aaad-44a6-bdf9-9a46893a16d7" containerName="payload-processing" containerID="cri-o://4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7" gracePeriod=25 Oct 30 06:23:57.363672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.363644 5001 generic.go:356] "Generic (PLEG): container finished" podID="81fb235e-78b9-4f68-8698-e62153540562" containerID="f8a8dc257d324a47bb2209c41250ffc8fdc0fe9189b0923eb654ac39003ea39b" exitCode=0 Oct 30 06:23:57.364074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.363787 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" event={"ID":"81fb235e-78b9-4f68-8698-e62153540562","Type":"ContainerDied","Data":"f8a8dc257d324a47bb2209c41250ffc8fdc0fe9189b0923eb654ac39003ea39b"} Oct 30 06:23:57.368575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.368545 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" podUID="e6486dc4-552e-4e15-a6f4-24228916eeaf" containerName="payload-pre-processing" containerID="cri-o://2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442" gracePeriod=25 Oct 30 06:23:57.782982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.782953 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:23:57.787976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.787951 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:23:57.884190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.884115 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42deeec4-aaad-44a6-bdf9-9a46893a16d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42deeec4-aaad-44a6-bdf9-9a46893a16d7-plugins-config-volume\") pod \"42deeec4-aaad-44a6-bdf9-9a46893a16d7\" (UID: \"42deeec4-aaad-44a6-bdf9-9a46893a16d7\") " Oct 30 06:23:57.884353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.884190 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6486dc4-552e-4e15-a6f4-24228916eeaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6486dc4-552e-4e15-a6f4-24228916eeaf-plugins-config-volume\") pod \"e6486dc4-552e-4e15-a6f4-24228916eeaf\" (UID: \"e6486dc4-552e-4e15-a6f4-24228916eeaf\") " Oct 30 06:23:57.884353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.884239 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6486dc4-552e-4e15-a6f4-24228916eeaf-kube-api-access-jnv9m\" (UniqueName: \"kubernetes.io/projected/e6486dc4-552e-4e15-a6f4-24228916eeaf-kube-api-access-jnv9m\") pod \"e6486dc4-552e-4e15-a6f4-24228916eeaf\" (UID: \"e6486dc4-552e-4e15-a6f4-24228916eeaf\") " Oct 30 06:23:57.884353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.884281 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42deeec4-aaad-44a6-bdf9-9a46893a16d7-kube-api-access-8mkqj\" (UniqueName: \"kubernetes.io/projected/42deeec4-aaad-44a6-bdf9-9a46893a16d7-kube-api-access-8mkqj\") pod \"42deeec4-aaad-44a6-bdf9-9a46893a16d7\" (UID: \"42deeec4-aaad-44a6-bdf9-9a46893a16d7\") " Oct 30 06:23:57.885529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.885503 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6486dc4-552e-4e15-a6f4-24228916eeaf-plugins-config-volume" pod "e6486dc4-552e-4e15-a6f4-24228916eeaf" (UID: "e6486dc4-552e-4e15-a6f4-24228916eeaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:57.885991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.885967 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42deeec4-aaad-44a6-bdf9-9a46893a16d7-plugins-config-volume" pod "42deeec4-aaad-44a6-bdf9-9a46893a16d7" (UID: "42deeec4-aaad-44a6-bdf9-9a46893a16d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:57.887585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.887560 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6486dc4-552e-4e15-a6f4-24228916eeaf-kube-api-access-jnv9m" pod "e6486dc4-552e-4e15-a6f4-24228916eeaf" (UID: "e6486dc4-552e-4e15-a6f4-24228916eeaf"). InnerVolumeSpecName "kube-api-access-jnv9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:57.887754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.887737 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42deeec4-aaad-44a6-bdf9-9a46893a16d7-kube-api-access-8mkqj" pod "42deeec4-aaad-44a6-bdf9-9a46893a16d7" (UID: "42deeec4-aaad-44a6-bdf9-9a46893a16d7"). InnerVolumeSpecName "kube-api-access-8mkqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:57.985927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.985892 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42deeec4-aaad-44a6-bdf9-9a46893a16d7-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.985927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.985925 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6486dc4-552e-4e15-a6f4-24228916eeaf-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.985927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.985934 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnv9m\" (UniqueName: \"kubernetes.io/projected/e6486dc4-552e-4e15-a6f4-24228916eeaf-kube-api-access-jnv9m\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.986163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:57.985944 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mkqj\" (UniqueName: \"kubernetes.io/projected/42deeec4-aaad-44a6-bdf9-9a46893a16d7-kube-api-access-8mkqj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:58.120551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.120519 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448e441c-2924-4232-93c1-721a7b633880" path="/var/lib/kubelet/pods/448e441c-2924-4232-93c1-721a7b633880/volumes" Oct 30 06:23:58.374093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.374060 5001 generic.go:356] "Generic (PLEG): container finished" podID="e6486dc4-552e-4e15-a6f4-24228916eeaf" containerID="2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442" exitCode=0 Oct 30 06:23:58.374571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.374114 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" Oct 30 06:23:58.374571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.374137 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" event={"ID":"e6486dc4-552e-4e15-a6f4-24228916eeaf","Type":"ContainerDied","Data":"2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442"} Oct 30 06:23:58.374571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.374209 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r" event={"ID":"e6486dc4-552e-4e15-a6f4-24228916eeaf","Type":"ContainerDied","Data":"097a3d74a94dffbd29b688080b3fc15ec98aa933e32e8e4470b5ac60f3d6a794"} Oct 30 06:23:58.374571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.374234 5001 scope.go:122] "RemoveContainer" containerID="2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442" Oct 30 06:23:58.376758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.376091 5001 generic.go:356] "Generic (PLEG): container finished" podID="42deeec4-aaad-44a6-bdf9-9a46893a16d7" containerID="4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7" exitCode=0 Oct 30 06:23:58.376758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.376123 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" event={"ID":"42deeec4-aaad-44a6-bdf9-9a46893a16d7","Type":"ContainerDied","Data":"4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7"} Oct 30 06:23:58.376758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.376190 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" event={"ID":"42deeec4-aaad-44a6-bdf9-9a46893a16d7","Type":"ContainerDied","Data":"b2c6a5ce493274e7f48a923a32f0fe1debfe87a14f11f047f6faa810700aa52b"} Oct 30 06:23:58.376758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.376155 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw" Oct 30 06:23:58.387089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.387065 5001 scope.go:122] "RemoveContainer" containerID="2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442" Oct 30 06:23:58.387626 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:23:58.387597 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442\": container with ID starting with 2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442 not found: ID does not exist" containerID="2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442" Oct 30 06:23:58.387772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.387636 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442"} err="failed to get container status \"2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442\": rpc error: code = NotFound desc = could not find container \"2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442\": container with ID starting with 2c801b3de2c04883e7fc4d1e24f68fe0beac4459532a6102a6f15885aaecf442 not found: ID does not exist" Oct 30 06:23:58.387772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.387658 5001 scope.go:122] "RemoveContainer" containerID="4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7" Oct 30 06:23:58.399807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.399787 5001 scope.go:122] "RemoveContainer" containerID="4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7" Oct 30 06:23:58.400121 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:23:58.400096 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7\": container with ID starting with 4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7 not found: ID does not exist" containerID="4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7" Oct 30 06:23:58.400176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.400131 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7"} err="failed to get container status \"4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7\": rpc error: code = NotFound desc = could not find container \"4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7\": container with ID starting with 4f41d18c1b1ede2edc75d102c948099bb54606c97a1af50754be0dad76011fd7 not found: ID does not exist" Oct 30 06:23:58.425687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.425660 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw"] Oct 30 06:23:58.435730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.435703 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93c22e-55546fbb88-9nmfw"] Oct 30 06:23:58.470486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.470455 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r"] Oct 30 06:23:58.473857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.473835 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93c22e-6dff459747-tfd4r"] Oct 30 06:23:58.549539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.549515 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:58.695244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.695162 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fb235e-78b9-4f68-8698-e62153540562-kube-api-access-6p4gl\" (UniqueName: \"kubernetes.io/projected/81fb235e-78b9-4f68-8698-e62153540562-kube-api-access-6p4gl\") pod \"81fb235e-78b9-4f68-8698-e62153540562\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " Oct 30 06:23:58.695440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.695260 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81fb235e-78b9-4f68-8698-e62153540562-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fb235e-78b9-4f68-8698-e62153540562-maas-api-openshift-ca\") pod \"81fb235e-78b9-4f68-8698-e62153540562\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " Oct 30 06:23:58.695440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.695386 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81fb235e-78b9-4f68-8698-e62153540562-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fb235e-78b9-4f68-8698-e62153540562-maas-api-xks-ca\") pod \"81fb235e-78b9-4f68-8698-e62153540562\" (UID: \"81fb235e-78b9-4f68-8698-e62153540562\") " Oct 30 06:23:58.695668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.695641 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fb235e-78b9-4f68-8698-e62153540562-maas-api-openshift-ca" pod "81fb235e-78b9-4f68-8698-e62153540562" (UID: "81fb235e-78b9-4f68-8698-e62153540562"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:58.695843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.695826 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fb235e-78b9-4f68-8698-e62153540562-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:58.697716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.697687 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fb235e-78b9-4f68-8698-e62153540562-kube-api-access-6p4gl" pod "81fb235e-78b9-4f68-8698-e62153540562" (UID: "81fb235e-78b9-4f68-8698-e62153540562"). InnerVolumeSpecName "kube-api-access-6p4gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:58.697843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.697764 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81fb235e-78b9-4f68-8698-e62153540562-maas-api-xks-ca" pod "81fb235e-78b9-4f68-8698-e62153540562" (UID: "81fb235e-78b9-4f68-8698-e62153540562"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:58.796796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.796763 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p4gl\" (UniqueName: \"kubernetes.io/projected/81fb235e-78b9-4f68-8698-e62153540562-kube-api-access-6p4gl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:58.796796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:58.796792 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fb235e-78b9-4f68-8698-e62153540562-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:23:59.388786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:59.388753 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" Oct 30 06:23:59.388786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:59.388776 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb" event={"ID":"81fb235e-78b9-4f68-8698-e62153540562","Type":"ContainerDied","Data":"670bb2799bd59711ad3bd38b3f6954d3d674b799b2c05e6c29c82d7354533874"} Oct 30 06:23:59.389475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:23:59.388827 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="670bb2799bd59711ad3bd38b3f6954d3d674b799b2c05e6c29c82d7354533874" Oct 30 06:24:00.118002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:00.117964 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42deeec4-aaad-44a6-bdf9-9a46893a16d7" path="/var/lib/kubelet/pods/42deeec4-aaad-44a6-bdf9-9a46893a16d7/volumes" Oct 30 06:24:00.118451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:00.118435 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6486dc4-552e-4e15-a6f4-24228916eeaf" path="/var/lib/kubelet/pods/e6486dc4-552e-4e15-a6f4-24228916eeaf/volumes" Oct 30 06:24:03.460411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:03.460372 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk"] Oct 30 06:24:03.461207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:03.461149 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" containerID="cri-o://f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168" gracePeriod=30 Oct 30 06:24:07.176515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.176478 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8"] Oct 30 06:24:07.177375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177353 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e6486dc4-552e-4e15-a6f4-24228916eeaf" containerName="payload-pre-processing" Oct 30 06:24:07.177466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177378 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6486dc4-552e-4e15-a6f4-24228916eeaf" containerName="payload-pre-processing" Oct 30 06:24:07.177466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177426 5001 cpu_manager.go:414] "containerMap: removing container" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" Oct 30 06:24:07.177466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177434 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" Oct 30 06:24:07.177571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177478 5001 cpu_manager.go:414] "containerMap: removing container" podUID="42deeec4-aaad-44a6-bdf9-9a46893a16d7" containerName="payload-processing" Oct 30 06:24:07.177571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177485 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42deeec4-aaad-44a6-bdf9-9a46893a16d7" containerName="payload-processing" Oct 30 06:24:07.177571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177496 5001 cpu_manager.go:414] "containerMap: removing container" podUID="81fb235e-78b9-4f68-8698-e62153540562" containerName="revoke-keys" Oct 30 06:24:07.177571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177503 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81fb235e-78b9-4f68-8698-e62153540562" containerName="revoke-keys" Oct 30 06:24:07.177571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177517 5001 cpu_manager.go:414] "containerMap: removing container" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" Oct 30 06:24:07.177571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177525 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" Oct 30 06:24:07.177759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177661 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" Oct 30 06:24:07.177759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177676 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="42deeec4-aaad-44a6-bdf9-9a46893a16d7" containerName="payload-processing" Oct 30 06:24:07.177759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177687 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="81fb235e-78b9-4f68-8698-e62153540562" containerName="revoke-keys" Oct 30 06:24:07.177759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.177700 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6486dc4-552e-4e15-a6f4-24228916eeaf" containerName="payload-pre-processing" Oct 30 06:24:07.183053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.183035 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:07.200378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.200350 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8"] Oct 30 06:24:07.206122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.206093 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:24:07.206277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.206176 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:24:07.206277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.206202 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:24:07.211791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.211768 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:24:07.216433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.216407 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g94nh\" (UniqueName: \"kubernetes.io/projected/afc1bfd1-6c91-484b-8caa-768fe57d18b9-kube-api-access-g94nh\") pod \"maas-api-revoke-sub-398718f0-jvzq8\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:07.216550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.216451 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-398718f0-jvzq8\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:07.216550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.216485 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-398718f0-jvzq8\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:07.317372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.317343 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-398718f0-jvzq8\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:07.317582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.317565 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g94nh\" (UniqueName: \"kubernetes.io/projected/afc1bfd1-6c91-484b-8caa-768fe57d18b9-kube-api-access-g94nh\") pod \"maas-api-revoke-sub-398718f0-jvzq8\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:07.317675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.317591 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-398718f0-jvzq8\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:07.318250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.318225 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-398718f0-jvzq8\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:07.320017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.319997 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-398718f0-jvzq8\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:07.341929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.341904 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g94nh\" (UniqueName: \"kubernetes.io/projected/afc1bfd1-6c91-484b-8caa-768fe57d18b9-kube-api-access-g94nh\") pod \"maas-api-revoke-sub-398718f0-jvzq8\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:07.498225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.498123 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:07.823245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:07.823220 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8"] Oct 30 06:24:07.825783 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:07.825745 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc1bfd1_6c91_484b_8caa_768fe57d18b9.slice/crio-63e382914784d9751b67649d7420441c0474ccbfbec2a8866e38adeb5e3f583e WatchSource:0}: Error finding container 63e382914784d9751b67649d7420441c0474ccbfbec2a8866e38adeb5e3f583e: Status 404 returned error can't find the container with id 63e382914784d9751b67649d7420441c0474ccbfbec2a8866e38adeb5e3f583e Oct 30 06:24:08.268549 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:24:08.268506 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:24:08.465199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.465162 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" event={"ID":"afc1bfd1-6c91-484b-8caa-768fe57d18b9","Type":"ContainerStarted","Data":"3f7af04cc1abf351ab09a2162d0353e41ce3e2e5200dc8384ab7f70caa8cd3e5"} Oct 30 06:24:08.465810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.465235 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" event={"ID":"afc1bfd1-6c91-484b-8caa-768fe57d18b9","Type":"ContainerStarted","Data":"63e382914784d9751b67649d7420441c0474ccbfbec2a8866e38adeb5e3f583e"} Oct 30 06:24:08.506204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.506149 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" podStartSLOduration=1.5061263710000001 podStartE2EDuration="1.506126371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:08.505216588 +0000 UTC m=+1862.501173599" watchObservedRunningTime="2026-10-30 06:24:08.506126371 +0000 UTC m=+1862.502083382" Oct 30 06:24:08.838903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.838836 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:24:08.970140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.970101 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-credential-socket\") pod \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " Oct 30 06:24:08.970373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.970175 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-data\") pod \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " Oct 30 06:24:08.970373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.970245 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istiod-ca-cert\") pod \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " Oct 30 06:24:08.970373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.970290 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-podinfo\") pod \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " Oct 30 06:24:08.970373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.970338 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-socket\") pod \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " Oct 30 06:24:08.970675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.970380 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-token\" (UniqueName: \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-token\") pod \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " Oct 30 06:24:08.970675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.970415 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-certs\") pod \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " Oct 30 06:24:08.970675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.970424 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-credential-socket" pod "eb19e5cd-434e-4d2d-bf69-ab0412742d3c" (UID: "eb19e5cd-434e-4d2d-bf69-ab0412742d3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.970675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.970471 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-kube-api-access-4mnqg\" (UniqueName: \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-kube-api-access-4mnqg\") pod \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " Oct 30 06:24:08.970675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.970521 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-envoy\") pod \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\" (UID: \"eb19e5cd-434e-4d2d-bf69-ab0412742d3c\") " Oct 30 06:24:08.971071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.971044 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.971849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.971820 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-socket" pod "eb19e5cd-434e-4d2d-bf69-ab0412742d3c" (UID: "eb19e5cd-434e-4d2d-bf69-ab0412742d3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.972050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.972003 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istiod-ca-cert" pod "eb19e5cd-434e-4d2d-bf69-ab0412742d3c" (UID: "eb19e5cd-434e-4d2d-bf69-ab0412742d3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:08.972153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.972124 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-data" pod "eb19e5cd-434e-4d2d-bf69-ab0412742d3c" (UID: "eb19e5cd-434e-4d2d-bf69-ab0412742d3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.972497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.972473 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-certs" pod "eb19e5cd-434e-4d2d-bf69-ab0412742d3c" (UID: "eb19e5cd-434e-4d2d-bf69-ab0412742d3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.973546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.973521 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-kube-api-access-4mnqg" pod "eb19e5cd-434e-4d2d-bf69-ab0412742d3c" (UID: "eb19e5cd-434e-4d2d-bf69-ab0412742d3c"). InnerVolumeSpecName "kube-api-access-4mnqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.973801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.973779 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-podinfo" pod "eb19e5cd-434e-4d2d-bf69-ab0412742d3c" (UID: "eb19e5cd-434e-4d2d-bf69-ab0412742d3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:08.973918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.973816 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-token" pod "eb19e5cd-434e-4d2d-bf69-ab0412742d3c" (UID: "eb19e5cd-434e-4d2d-bf69-ab0412742d3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.973918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:08.973827 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-envoy" pod "eb19e5cd-434e-4d2d-bf69-ab0412742d3c" (UID: "eb19e5cd-434e-4d2d-bf69-ab0412742d3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:09.076352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.075262 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.076352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.075329 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.076352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.075345 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mnqg\" (UniqueName: \"kubernetes.io/projected/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-kube-api-access-4mnqg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.076352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.075360 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.076352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.075374 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.076352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.075386 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.076352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.075399 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.076352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.075411 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb19e5cd-434e-4d2d-bf69-ab0412742d3c-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.475206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.475167 5001 generic.go:356] "Generic (PLEG): container finished" podID="afc1bfd1-6c91-484b-8caa-768fe57d18b9" containerID="3f7af04cc1abf351ab09a2162d0353e41ce3e2e5200dc8384ab7f70caa8cd3e5" exitCode=0 Oct 30 06:24:09.475678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.475263 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" event={"ID":"afc1bfd1-6c91-484b-8caa-768fe57d18b9","Type":"ContainerDied","Data":"3f7af04cc1abf351ab09a2162d0353e41ce3e2e5200dc8384ab7f70caa8cd3e5"} Oct 30 06:24:09.477162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.477139 5001 generic.go:356] "Generic (PLEG): container finished" podID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerID="f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168" exitCode=0 Oct 30 06:24:09.477338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.477197 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" Oct 30 06:24:09.477338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.477210 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" event={"ID":"eb19e5cd-434e-4d2d-bf69-ab0412742d3c","Type":"ContainerDied","Data":"f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168"} Oct 30 06:24:09.477338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.477258 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk" event={"ID":"eb19e5cd-434e-4d2d-bf69-ab0412742d3c","Type":"ContainerDied","Data":"dd02b27b3c27017d01c5db7c2c922ea8ae1d8d679817e2ef4ce0c8289ccc3d9b"} Oct 30 06:24:09.477338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.477287 5001 scope.go:122] "RemoveContainer" containerID="f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168" Oct 30 06:24:09.490675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.490645 5001 scope.go:122] "RemoveContainer" containerID="8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3" Oct 30 06:24:09.500110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.500092 5001 scope.go:122] "RemoveContainer" containerID="f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168" Oct 30 06:24:09.500409 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:24:09.500388 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168\": container with ID starting with f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168 not found: ID does not exist" containerID="f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168" Oct 30 06:24:09.500471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.500417 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168"} err="failed to get container status \"f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168\": rpc error: code = NotFound desc = could not find container \"f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168\": container with ID starting with f9024404b683f1a5f10b45f5b70e34efcf2df35e0d41bb536ed29193be59a168 not found: ID does not exist" Oct 30 06:24:09.500471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.500434 5001 scope.go:122] "RemoveContainer" containerID="8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3" Oct 30 06:24:09.500691 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:24:09.500674 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3\": container with ID starting with 8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3 not found: ID does not exist" containerID="8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3" Oct 30 06:24:09.500736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.500698 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3"} err="failed to get container status \"8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3\": rpc error: code = NotFound desc = could not find container \"8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3\": container with ID starting with 8e5be422c837b37a08022e27c1bbdb808dd301b247683297546ba763857c67a3 not found: ID does not exist" Oct 30 06:24:09.552687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.552657 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk"] Oct 30 06:24:09.565359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:09.565325 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-4dd310-openshift-default-7ffc75bf7c-l2dkk"] Oct 30 06:24:10.118042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.118011 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" path="/var/lib/kubelet/pods/eb19e5cd-434e-4d2d-bf69-ab0412742d3c/volumes" Oct 30 06:24:10.505147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.505108 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5"] Oct 30 06:24:10.505980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.505955 5001 cpu_manager.go:414] "containerMap: removing container" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" Oct 30 06:24:10.505980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.505977 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" Oct 30 06:24:10.506191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.506164 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="448e441c-2924-4232-93c1-721a7b633880" containerName="istio-proxy" Oct 30 06:24:10.506191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.506179 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" Oct 30 06:24:10.506299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.506192 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" Oct 30 06:24:10.511813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.511790 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:10.527884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.527851 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5"] Oct 30 06:24:10.593085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.593044 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb3d825-6kzk5\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:10.593262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.593101 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkrxj\" (UniqueName: \"kubernetes.io/projected/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-kube-api-access-nkrxj\") pod \"maas-api-revoke-sub-bdb3d825-6kzk5\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:10.593450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.593422 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb3d825-6kzk5\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:10.679346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.679197 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:10.694365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.694334 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-openshift-ca\") pod \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " Oct 30 06:24:10.694546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.694396 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-xks-ca\") pod \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " Oct 30 06:24:10.694615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.694577 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc1bfd1-6c91-484b-8caa-768fe57d18b9-kube-api-access-g94nh\" (UniqueName: \"kubernetes.io/projected/afc1bfd1-6c91-484b-8caa-768fe57d18b9-kube-api-access-g94nh\") pod \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\" (UID: \"afc1bfd1-6c91-484b-8caa-768fe57d18b9\") " Oct 30 06:24:10.694681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.694649 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-openshift-ca" pod "afc1bfd1-6c91-484b-8caa-768fe57d18b9" (UID: "afc1bfd1-6c91-484b-8caa-768fe57d18b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:10.694796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.694777 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb3d825-6kzk5\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:10.694856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.694809 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkrxj\" (UniqueName: \"kubernetes.io/projected/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-kube-api-access-nkrxj\") pod \"maas-api-revoke-sub-bdb3d825-6kzk5\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:10.694974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.694959 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb3d825-6kzk5\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:10.695179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.695163 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:10.695590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.695565 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb3d825-6kzk5\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:10.696728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.696702 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-xks-ca" pod "afc1bfd1-6c91-484b-8caa-768fe57d18b9" (UID: "afc1bfd1-6c91-484b-8caa-768fe57d18b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:10.696900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.696879 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc1bfd1-6c91-484b-8caa-768fe57d18b9-kube-api-access-g94nh" pod "afc1bfd1-6c91-484b-8caa-768fe57d18b9" (UID: "afc1bfd1-6c91-484b-8caa-768fe57d18b9"). InnerVolumeSpecName "kube-api-access-g94nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:10.697457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.697437 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb3d825-6kzk5\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:10.738158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.727239 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkrxj\" (UniqueName: \"kubernetes.io/projected/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-kube-api-access-nkrxj\") pod \"maas-api-revoke-sub-bdb3d825-6kzk5\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:10.796054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.796020 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc1bfd1-6c91-484b-8caa-768fe57d18b9-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:10.796054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.796050 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g94nh\" (UniqueName: \"kubernetes.io/projected/afc1bfd1-6c91-484b-8caa-768fe57d18b9-kube-api-access-g94nh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:10.832232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:10.832197 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:11.137673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:11.137648 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5"] Oct 30 06:24:11.140297 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:11.140258 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d24bf2c_e5ce_40ea_b861_8aa56e57603f.slice/crio-f23fe5eb0a3833d5a7146f6e506e7e0999212e238d24eac7e967f46c8a881c18 WatchSource:0}: Error finding container f23fe5eb0a3833d5a7146f6e506e7e0999212e238d24eac7e967f46c8a881c18: Status 404 returned error can't find the container with id f23fe5eb0a3833d5a7146f6e506e7e0999212e238d24eac7e967f46c8a881c18 Oct 30 06:24:11.495932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:11.495893 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" Oct 30 06:24:11.495932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:11.495918 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8" event={"ID":"afc1bfd1-6c91-484b-8caa-768fe57d18b9","Type":"ContainerDied","Data":"63e382914784d9751b67649d7420441c0474ccbfbec2a8866e38adeb5e3f583e"} Oct 30 06:24:11.496168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:11.495969 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63e382914784d9751b67649d7420441c0474ccbfbec2a8866e38adeb5e3f583e" Oct 30 06:24:11.497511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:11.497471 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" event={"ID":"0d24bf2c-e5ce-40ea-b861-8aa56e57603f","Type":"ContainerStarted","Data":"f23fe5eb0a3833d5a7146f6e506e7e0999212e238d24eac7e967f46c8a881c18"} Oct 30 06:24:11.502418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:11.502393 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976"] Oct 30 06:24:11.513294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:11.513270 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c22e-4m976"] Oct 30 06:24:12.118489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:12.118400 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5d5603-8fed-47e4-9589-a5caa4f978e7" path="/var/lib/kubelet/pods/ac5d5603-8fed-47e4-9589-a5caa4f978e7/volumes" Oct 30 06:24:12.507293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:12.507214 5001 generic.go:356] "Generic (PLEG): container finished" podID="0d24bf2c-e5ce-40ea-b861-8aa56e57603f" containerID="c76bc7ad2e5817141da9f96490d1ebf32a5458a397c40683015acc9d699de800" exitCode=0 Oct 30 06:24:12.507463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:12.507293 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" event={"ID":"0d24bf2c-e5ce-40ea-b861-8aa56e57603f","Type":"ContainerDied","Data":"c76bc7ad2e5817141da9f96490d1ebf32a5458a397c40683015acc9d699de800"} Oct 30 06:24:13.693832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:13.693805 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:13.757924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:13.757896 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-openshift-ca\") pod \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " Oct 30 06:24:13.758124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:13.758015 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-xks-ca\") pod \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " Oct 30 06:24:13.758124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:13.758049 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-kube-api-access-nkrxj\" (UniqueName: \"kubernetes.io/projected/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-kube-api-access-nkrxj\") pod \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\" (UID: \"0d24bf2c-e5ce-40ea-b861-8aa56e57603f\") " Oct 30 06:24:13.758492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:13.758461 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-openshift-ca" pod "0d24bf2c-e5ce-40ea-b861-8aa56e57603f" (UID: "0d24bf2c-e5ce-40ea-b861-8aa56e57603f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:13.758780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:13.758757 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:13.761435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:13.761407 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-xks-ca" pod "0d24bf2c-e5ce-40ea-b861-8aa56e57603f" (UID: "0d24bf2c-e5ce-40ea-b861-8aa56e57603f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:13.764138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:13.764110 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-kube-api-access-nkrxj" pod "0d24bf2c-e5ce-40ea-b861-8aa56e57603f" (UID: "0d24bf2c-e5ce-40ea-b861-8aa56e57603f"). InnerVolumeSpecName "kube-api-access-nkrxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:13.860424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:13.860385 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:13.860424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:13.860421 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkrxj\" (UniqueName: \"kubernetes.io/projected/0d24bf2c-e5ce-40ea-b861-8aa56e57603f-kube-api-access-nkrxj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:14.526640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:14.526553 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" event={"ID":"0d24bf2c-e5ce-40ea-b861-8aa56e57603f","Type":"ContainerDied","Data":"f23fe5eb0a3833d5a7146f6e506e7e0999212e238d24eac7e967f46c8a881c18"} Oct 30 06:24:14.526640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:14.526593 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5" Oct 30 06:24:14.526640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:14.526606 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f23fe5eb0a3833d5a7146f6e506e7e0999212e238d24eac7e967f46c8a881c18" Oct 30 06:24:19.623359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:19.623294 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg"] Oct 30 06:24:19.623858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:19.623613 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" containerID="cri-o://17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff" gracePeriod=30 Oct 30 06:24:20.776127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.776085 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4"] Oct 30 06:24:20.776728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.776711 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0d24bf2c-e5ce-40ea-b861-8aa56e57603f" containerName="revoke-keys" Oct 30 06:24:20.776728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.776728 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d24bf2c-e5ce-40ea-b861-8aa56e57603f" containerName="revoke-keys" Oct 30 06:24:20.776851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.776747 5001 cpu_manager.go:414] "containerMap: removing container" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" Oct 30 06:24:20.776851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.776752 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb19e5cd-434e-4d2d-bf69-ab0412742d3c" containerName="istio-proxy" Oct 30 06:24:20.776851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.776762 5001 cpu_manager.go:414] "containerMap: removing container" podUID="afc1bfd1-6c91-484b-8caa-768fe57d18b9" containerName="revoke-keys" Oct 30 06:24:20.776851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.776767 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc1bfd1-6c91-484b-8caa-768fe57d18b9" containerName="revoke-keys" Oct 30 06:24:20.776851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.776844 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d24bf2c-e5ce-40ea-b861-8aa56e57603f" containerName="revoke-keys" Oct 30 06:24:20.777141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.776856 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc1bfd1-6c91-484b-8caa-768fe57d18b9" containerName="revoke-keys" Oct 30 06:24:20.780791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.780766 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:20.808876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.808798 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:24:20.809169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.809145 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:24:20.809466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.809447 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:24:20.809637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.809622 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:24:20.815166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.815134 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4"] Oct 30 06:24:20.881437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.881398 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb32039-s55w4\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:20.881639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.881579 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvp6c\" (UniqueName: \"kubernetes.io/projected/e274757d-d67e-4857-8dee-db81b94c65a7-kube-api-access-hvp6c\") pod \"maas-api-revoke-sub-bbb32039-s55w4\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:20.881729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.881710 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb32039-s55w4\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:20.982907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.982864 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb32039-s55w4\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:20.983098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.982989 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb32039-s55w4\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:20.983148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.983124 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvp6c\" (UniqueName: \"kubernetes.io/projected/e274757d-d67e-4857-8dee-db81b94c65a7-kube-api-access-hvp6c\") pod \"maas-api-revoke-sub-bbb32039-s55w4\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:20.983817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.983734 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb32039-s55w4\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:20.985710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:20.985685 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb32039-s55w4\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:21.006544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:21.006520 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvp6c\" (UniqueName: \"kubernetes.io/projected/e274757d-d67e-4857-8dee-db81b94c65a7-kube-api-access-hvp6c\") pod \"maas-api-revoke-sub-bbb32039-s55w4\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:21.095125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:21.095030 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:21.416464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:21.416439 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4"] Oct 30 06:24:21.418986 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:21.418953 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode274757d_d67e_4857_8dee_db81b94c65a7.slice/crio-2ca4a84e27c881906871563a712348097bc971b87c52388657e4c80d82943ba3 WatchSource:0}: Error finding container 2ca4a84e27c881906871563a712348097bc971b87c52388657e4c80d82943ba3: Status 404 returned error can't find the container with id 2ca4a84e27c881906871563a712348097bc971b87c52388657e4c80d82943ba3 Oct 30 06:24:21.587972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:21.587937 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" event={"ID":"e274757d-d67e-4857-8dee-db81b94c65a7","Type":"ContainerStarted","Data":"2ca4a84e27c881906871563a712348097bc971b87c52388657e4c80d82943ba3"} Oct 30 06:24:22.074666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:22.074574 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:24:22.597633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:22.597597 5001 generic.go:356] "Generic (PLEG): container finished" podID="e274757d-d67e-4857-8dee-db81b94c65a7" containerID="8df8a01e78de008ba032dcf64faccd071925dc3e858f4e432b711168992cf72b" exitCode=0 Oct 30 06:24:22.597805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:22.597683 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" event={"ID":"e274757d-d67e-4857-8dee-db81b94c65a7","Type":"ContainerDied","Data":"8df8a01e78de008ba032dcf64faccd071925dc3e858f4e432b711168992cf72b"} Oct 30 06:24:23.768503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.768473 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7"] Oct 30 06:24:23.772905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.772882 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:23.794800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.794768 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7"] Oct 30 06:24:23.800401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.800364 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:23.930439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.930356 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e274757d-d67e-4857-8dee-db81b94c65a7-kube-api-access-hvp6c\" (UniqueName: \"kubernetes.io/projected/e274757d-d67e-4857-8dee-db81b94c65a7-kube-api-access-hvp6c\") pod \"e274757d-d67e-4857-8dee-db81b94c65a7\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " Oct 30 06:24:23.930588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.930494 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-openshift-ca\") pod \"e274757d-d67e-4857-8dee-db81b94c65a7\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " Oct 30 06:24:23.930588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.930525 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-xks-ca\") pod \"e274757d-d67e-4857-8dee-db81b94c65a7\" (UID: \"e274757d-d67e-4857-8dee-db81b94c65a7\") " Oct 30 06:24:23.930758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.930741 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38b19e07-kzxl7\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:23.930849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.930785 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38b19e07-kzxl7\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:23.930908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.930829 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-openshift-ca" pod "e274757d-d67e-4857-8dee-db81b94c65a7" (UID: "e274757d-d67e-4857-8dee-db81b94c65a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:23.930969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.930915 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgvv\" (UniqueName: \"kubernetes.io/projected/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-kube-api-access-5fgvv\") pod \"maas-api-revoke-sub-38b19e07-kzxl7\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:23.931072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.931051 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:23.932777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.932754 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e274757d-d67e-4857-8dee-db81b94c65a7-kube-api-access-hvp6c" pod "e274757d-d67e-4857-8dee-db81b94c65a7" (UID: "e274757d-d67e-4857-8dee-db81b94c65a7"). InnerVolumeSpecName "kube-api-access-hvp6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:23.932871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:23.932805 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-xks-ca" pod "e274757d-d67e-4857-8dee-db81b94c65a7" (UID: "e274757d-d67e-4857-8dee-db81b94c65a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:24.032478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.032439 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38b19e07-kzxl7\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:24.032714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.032496 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38b19e07-kzxl7\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:24.032714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.032540 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgvv\" (UniqueName: \"kubernetes.io/projected/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-kube-api-access-5fgvv\") pod \"maas-api-revoke-sub-38b19e07-kzxl7\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:24.032714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.032630 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvp6c\" (UniqueName: \"kubernetes.io/projected/e274757d-d67e-4857-8dee-db81b94c65a7-kube-api-access-hvp6c\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:24.032714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.032642 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e274757d-d67e-4857-8dee-db81b94c65a7-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:24.033205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.033182 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38b19e07-kzxl7\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:24.034967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.034944 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38b19e07-kzxl7\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:24.056321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.056276 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgvv\" (UniqueName: \"kubernetes.io/projected/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-kube-api-access-5fgvv\") pod \"maas-api-revoke-sub-38b19e07-kzxl7\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:24.113551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.113503 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:24.449379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.443737 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km"] Oct 30 06:24:24.449379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.444000 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" containerID="cri-o://e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd" gracePeriod=30 Oct 30 06:24:24.483819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.483786 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7"] Oct 30 06:24:24.487737 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:24.487713 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15722fe8_c60d_4e9d_b5c2_bbe3b7b49947.slice/crio-64de8237dff4676516699e438d9c56e06312b2fe88941969bbf0e677f65f4dd5 WatchSource:0}: Error finding container 64de8237dff4676516699e438d9c56e06312b2fe88941969bbf0e677f65f4dd5: Status 404 returned error can't find the container with id 64de8237dff4676516699e438d9c56e06312b2fe88941969bbf0e677f65f4dd5 Oct 30 06:24:24.663568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.663037 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" event={"ID":"e274757d-d67e-4857-8dee-db81b94c65a7","Type":"ContainerDied","Data":"2ca4a84e27c881906871563a712348097bc971b87c52388657e4c80d82943ba3"} Oct 30 06:24:24.663568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.663100 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ca4a84e27c881906871563a712348097bc971b87c52388657e4c80d82943ba3" Oct 30 06:24:24.663568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.663179 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4" Oct 30 06:24:24.667953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.666202 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" event={"ID":"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947","Type":"ContainerStarted","Data":"64de8237dff4676516699e438d9c56e06312b2fe88941969bbf0e677f65f4dd5"} Oct 30 06:24:24.967399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:24.967370 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:24:25.111725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.111695 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn"] Oct 30 06:24:25.112293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.112279 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e274757d-d67e-4857-8dee-db81b94c65a7" containerName="revoke-keys" Oct 30 06:24:25.112372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.112296 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e274757d-d67e-4857-8dee-db81b94c65a7" containerName="revoke-keys" Oct 30 06:24:25.112372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.112336 5001 cpu_manager.go:414] "containerMap: removing container" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" Oct 30 06:24:25.112372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.112343 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" Oct 30 06:24:25.112372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.112363 5001 cpu_manager.go:414] "containerMap: removing container" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" Oct 30 06:24:25.112372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.112368 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" Oct 30 06:24:25.112532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.112471 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" Oct 30 06:24:25.112532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.112485 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e274757d-d67e-4857-8dee-db81b94c65a7" containerName="revoke-keys" Oct 30 06:24:25.112532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.112494 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerName="istio-proxy" Oct 30 06:24:25.116029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.116011 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:25.140814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.140781 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn"] Oct 30 06:24:25.155329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.153367 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-envoy\") pod \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " Oct 30 06:24:25.155329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.153432 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-credential-socket\") pod \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " Oct 30 06:24:25.155329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.153503 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-certs\") pod \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " Oct 30 06:24:25.155329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.153557 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-token\" (UniqueName: \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-token\") pod \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " Oct 30 06:24:25.155329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.153596 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-podinfo\") pod \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " Oct 30 06:24:25.155329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.153652 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-data\") pod \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " Oct 30 06:24:25.155329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.153702 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-socket\") pod \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " Oct 30 06:24:25.155329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.153749 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-credential-socket" pod "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" (UID: "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:25.155329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.154057 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-data" pod "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" (UID: "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:25.155329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.154557 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-certs" pod "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" (UID: "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:25.156184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.156117 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-socket" pod "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" (UID: "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:25.156656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.156628 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istiod-ca-cert" pod "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" (UID: "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:25.156763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.153818 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istiod-ca-cert\") pod \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " Oct 30 06:24:25.156886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.156781 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-kube-api-access-2vcbf\" (UniqueName: \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-kube-api-access-2vcbf\") pod \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\" (UID: \"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75\") " Oct 30 06:24:25.157691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.157667 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:25.157796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.157697 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:25.157796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.157711 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:25.157796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.157724 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:25.157796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.157737 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:25.160999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.160973 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-kube-api-access-2vcbf" pod "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" (UID: "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75"). InnerVolumeSpecName "kube-api-access-2vcbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:25.161936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.161911 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-token" pod "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" (UID: "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:25.163694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.163668 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-podinfo" pod "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" (UID: "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:25.164219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.164194 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-envoy" pod "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" (UID: "26ac07ad-23ba-47a7-bf7f-4bcd9315ff75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:25.259330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.258749 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:25.259330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.258896 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:25.259330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.259110 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9xwb\" (UniqueName: \"kubernetes.io/projected/37065385-a554-4c6f-89a7-416395f6c02b-kube-api-access-l9xwb\") pod \"maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:25.259662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.259435 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vcbf\" (UniqueName: \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-kube-api-access-2vcbf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:25.259662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.259466 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:25.259662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.259495 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:25.259662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.259510 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:25.360863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.360827 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:25.361067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.360889 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:25.361067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.361004 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9xwb\" (UniqueName: \"kubernetes.io/projected/37065385-a554-4c6f-89a7-416395f6c02b-kube-api-access-l9xwb\") pod \"maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:25.364113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.364031 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:25.364761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.364740 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:25.384110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.384078 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9xwb\" (UniqueName: \"kubernetes.io/projected/37065385-a554-4c6f-89a7-416395f6c02b-kube-api-access-l9xwb\") pod \"maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:25.429394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.429365 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:25.551589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.551550 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg"] Oct 30 06:24:25.556642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.556046 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:25.563065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.563037 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtknx\" (UniqueName: \"kubernetes.io/projected/d68bc927-978e-47f9-8510-31a6e25a83d7-kube-api-access-gtknx\") pod \"maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:25.563364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.563297 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:25.563517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.563446 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:25.586174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.586141 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg"] Oct 30 06:24:25.665226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.665133 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:25.665226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.665196 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:25.665499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.665296 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtknx\" (UniqueName: \"kubernetes.io/projected/d68bc927-978e-47f9-8510-31a6e25a83d7-kube-api-access-gtknx\") pod \"maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:25.666039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.666012 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:25.667899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.667870 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:25.677121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.677080 5001 generic.go:356] "Generic (PLEG): container finished" podID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" containerID="17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff" exitCode=0 Oct 30 06:24:25.677217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.677149 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" Oct 30 06:24:25.677217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.677158 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" event={"ID":"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75","Type":"ContainerDied","Data":"17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff"} Oct 30 06:24:25.677364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.677227 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg" event={"ID":"26ac07ad-23ba-47a7-bf7f-4bcd9315ff75","Type":"ContainerDied","Data":"ca8e1d26ecebb696c10505982f6a648ada3d553cf8d25fd583010b14b2ae6d89"} Oct 30 06:24:25.677364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.677259 5001 scope.go:122] "RemoveContainer" containerID="17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff" Oct 30 06:24:25.680032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.680001 5001 generic.go:356] "Generic (PLEG): container finished" podID="15722fe8-c60d-4e9d-b5c2-bbe3b7b49947" containerID="e3e2e23db723f7ef724a8ddc2042a8c8659a250685a27759daab5380621a3db9" exitCode=0 Oct 30 06:24:25.680233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.680215 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" event={"ID":"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947","Type":"ContainerDied","Data":"e3e2e23db723f7ef724a8ddc2042a8c8659a250685a27759daab5380621a3db9"} Oct 30 06:24:25.688600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.688579 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtknx\" (UniqueName: \"kubernetes.io/projected/d68bc927-978e-47f9-8510-31a6e25a83d7-kube-api-access-gtknx\") pod \"maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:25.690673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.690651 5001 scope.go:122] "RemoveContainer" containerID="e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89" Oct 30 06:24:25.702425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.702405 5001 scope.go:122] "RemoveContainer" containerID="17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff" Oct 30 06:24:25.702732 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:24:25.702710 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff\": container with ID starting with 17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff not found: ID does not exist" containerID="17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff" Oct 30 06:24:25.702814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.702741 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff"} err="failed to get container status \"17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff\": rpc error: code = NotFound desc = could not find container \"17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff\": container with ID starting with 17296db7561419d0af56f0c0107bd514d7796f341e4d18717019aa9a4047a0ff not found: ID does not exist" Oct 30 06:24:25.702814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.702757 5001 scope.go:122] "RemoveContainer" containerID="e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89" Oct 30 06:24:25.703021 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:24:25.703002 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89\": container with ID starting with e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89 not found: ID does not exist" containerID="e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89" Oct 30 06:24:25.703075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.703027 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89"} err="failed to get container status \"e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89\": rpc error: code = NotFound desc = could not find container \"e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89\": container with ID starting with e3ce895a585d4c6d490cb5abe74541e8cca3fb211eace3fff354e0d10a8bec89 not found: ID does not exist" Oct 30 06:24:25.746325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.746286 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn"] Oct 30 06:24:25.748726 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:25.748686 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37065385_a554_4c6f_89a7_416395f6c02b.slice/crio-6d6713314e82cc8efd36e7c4e93193d0abe8b6afa83eaf99715d03623b41631d WatchSource:0}: Error finding container 6d6713314e82cc8efd36e7c4e93193d0abe8b6afa83eaf99715d03623b41631d: Status 404 returned error can't find the container with id 6d6713314e82cc8efd36e7c4e93193d0abe8b6afa83eaf99715d03623b41631d Oct 30 06:24:25.772942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.772913 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg"] Oct 30 06:24:25.779129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.779101 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-93c22e-openshift-default-6f6d74d55c-5nmfg"] Oct 30 06:24:25.896936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:25.896907 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:26.120241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.120211 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ac07ad-23ba-47a7-bf7f-4bcd9315ff75" path="/var/lib/kubelet/pods/26ac07ad-23ba-47a7-bf7f-4bcd9315ff75/volumes" Oct 30 06:24:26.212899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.212829 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg"] Oct 30 06:24:26.216539 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:26.216495 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68bc927_978e_47f9_8510_31a6e25a83d7.slice/crio-b9f0d482917f1de7df861b51cb10ecbfb4e89d4d8de1ab2393b7237d47c5d6ae WatchSource:0}: Error finding container b9f0d482917f1de7df861b51cb10ecbfb4e89d4d8de1ab2393b7237d47c5d6ae: Status 404 returned error can't find the container with id b9f0d482917f1de7df861b51cb10ecbfb4e89d4d8de1ab2393b7237d47c5d6ae Oct 30 06:24:26.693604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.693566 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" event={"ID":"d68bc927-978e-47f9-8510-31a6e25a83d7","Type":"ContainerStarted","Data":"7027c571628cbe6be536982e6525334f026ff78c0c99a4c9582175f9fbe14c5f"} Oct 30 06:24:26.693791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.693628 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" event={"ID":"d68bc927-978e-47f9-8510-31a6e25a83d7","Type":"ContainerStarted","Data":"b9f0d482917f1de7df861b51cb10ecbfb4e89d4d8de1ab2393b7237d47c5d6ae"} Oct 30 06:24:26.695457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.695429 5001 generic.go:356] "Generic (PLEG): container finished" podID="37065385-a554-4c6f-89a7-416395f6c02b" containerID="cf62ea56c177222b92ddfae09c62a70724692e59ff7d18711e1a13847f065a86" exitCode=0 Oct 30 06:24:26.695604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.695499 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" event={"ID":"37065385-a554-4c6f-89a7-416395f6c02b","Type":"ContainerDied","Data":"cf62ea56c177222b92ddfae09c62a70724692e59ff7d18711e1a13847f065a86"} Oct 30 06:24:26.695604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.695561 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" event={"ID":"37065385-a554-4c6f-89a7-416395f6c02b","Type":"ContainerStarted","Data":"6d6713314e82cc8efd36e7c4e93193d0abe8b6afa83eaf99715d03623b41631d"} Oct 30 06:24:26.734595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.734485 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" podStartSLOduration=1.7344707050000001 podStartE2EDuration="1.734470705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:26.733427636 +0000 UTC m=+1880.729384648" watchObservedRunningTime="2026-10-30 06:24:26.734470705 +0000 UTC m=+1880.730427715" Oct 30 06:24:26.862971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.862942 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:26.887575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.887541 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-kube-api-access-5fgvv\" (UniqueName: \"kubernetes.io/projected/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-kube-api-access-5fgvv\") pod \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " Oct 30 06:24:26.887755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.887595 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-xks-ca\") pod \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " Oct 30 06:24:26.887823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.887807 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-openshift-ca\") pod \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\" (UID: \"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947\") " Oct 30 06:24:26.888402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.888284 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-openshift-ca" pod "15722fe8-c60d-4e9d-b5c2-bbe3b7b49947" (UID: "15722fe8-c60d-4e9d-b5c2-bbe3b7b49947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:26.890588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.890563 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-xks-ca" pod "15722fe8-c60d-4e9d-b5c2-bbe3b7b49947" (UID: "15722fe8-c60d-4e9d-b5c2-bbe3b7b49947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:26.890588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.890576 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-kube-api-access-5fgvv" pod "15722fe8-c60d-4e9d-b5c2-bbe3b7b49947" (UID: "15722fe8-c60d-4e9d-b5c2-bbe3b7b49947"). InnerVolumeSpecName "kube-api-access-5fgvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:26.989298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.989198 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:26.989298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.989240 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgvv\" (UniqueName: \"kubernetes.io/projected/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-kube-api-access-5fgvv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:26.989298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:26.989255 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:27.704489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.704458 5001 generic.go:356] "Generic (PLEG): container finished" podID="d68bc927-978e-47f9-8510-31a6e25a83d7" containerID="7027c571628cbe6be536982e6525334f026ff78c0c99a4c9582175f9fbe14c5f" exitCode=0 Oct 30 06:24:27.704956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.704533 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" event={"ID":"d68bc927-978e-47f9-8510-31a6e25a83d7","Type":"ContainerDied","Data":"7027c571628cbe6be536982e6525334f026ff78c0c99a4c9582175f9fbe14c5f"} Oct 30 06:24:27.706096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.706079 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" Oct 30 06:24:27.706207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.706115 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7" event={"ID":"15722fe8-c60d-4e9d-b5c2-bbe3b7b49947","Type":"ContainerDied","Data":"64de8237dff4676516699e438d9c56e06312b2fe88941969bbf0e677f65f4dd5"} Oct 30 06:24:27.706207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.706159 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64de8237dff4676516699e438d9c56e06312b2fe88941969bbf0e677f65f4dd5" Oct 30 06:24:27.871112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.871079 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:27.899511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.899480 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37065385-a554-4c6f-89a7-416395f6c02b-kube-api-access-l9xwb\" (UniqueName: \"kubernetes.io/projected/37065385-a554-4c6f-89a7-416395f6c02b-kube-api-access-l9xwb\") pod \"37065385-a554-4c6f-89a7-416395f6c02b\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " Oct 30 06:24:27.899692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.899535 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-openshift-ca\") pod \"37065385-a554-4c6f-89a7-416395f6c02b\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " Oct 30 06:24:27.899692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.899640 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-xks-ca\") pod \"37065385-a554-4c6f-89a7-416395f6c02b\" (UID: \"37065385-a554-4c6f-89a7-416395f6c02b\") " Oct 30 06:24:27.899989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.899961 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-openshift-ca" pod "37065385-a554-4c6f-89a7-416395f6c02b" (UID: "37065385-a554-4c6f-89a7-416395f6c02b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:27.900114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.900088 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:27.901875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.901844 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-xks-ca" pod "37065385-a554-4c6f-89a7-416395f6c02b" (UID: "37065385-a554-4c6f-89a7-416395f6c02b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:27.902051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:27.902033 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37065385-a554-4c6f-89a7-416395f6c02b-kube-api-access-l9xwb" pod "37065385-a554-4c6f-89a7-416395f6c02b" (UID: "37065385-a554-4c6f-89a7-416395f6c02b"). InnerVolumeSpecName "kube-api-access-l9xwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:28.001677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:28.001584 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37065385-a554-4c6f-89a7-416395f6c02b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:28.001677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:28.001614 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9xwb\" (UniqueName: \"kubernetes.io/projected/37065385-a554-4c6f-89a7-416395f6c02b-kube-api-access-l9xwb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:28.722810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:28.722773 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" Oct 30 06:24:28.723277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:28.722809 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn" event={"ID":"37065385-a554-4c6f-89a7-416395f6c02b","Type":"ContainerDied","Data":"6d6713314e82cc8efd36e7c4e93193d0abe8b6afa83eaf99715d03623b41631d"} Oct 30 06:24:28.723277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:28.722898 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d6713314e82cc8efd36e7c4e93193d0abe8b6afa83eaf99715d03623b41631d" Oct 30 06:24:28.912802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:28.912360 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:29.016167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.016083 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68bc927-978e-47f9-8510-31a6e25a83d7-kube-api-access-gtknx\" (UniqueName: \"kubernetes.io/projected/d68bc927-978e-47f9-8510-31a6e25a83d7-kube-api-access-gtknx\") pod \"d68bc927-978e-47f9-8510-31a6e25a83d7\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " Oct 30 06:24:29.016331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.016242 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-openshift-ca\") pod \"d68bc927-978e-47f9-8510-31a6e25a83d7\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " Oct 30 06:24:29.016383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.016350 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-xks-ca\") pod \"d68bc927-978e-47f9-8510-31a6e25a83d7\" (UID: \"d68bc927-978e-47f9-8510-31a6e25a83d7\") " Oct 30 06:24:29.016622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.016599 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-openshift-ca" pod "d68bc927-978e-47f9-8510-31a6e25a83d7" (UID: "d68bc927-978e-47f9-8510-31a6e25a83d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:29.016840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.016810 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.018556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.018534 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-xks-ca" pod "d68bc927-978e-47f9-8510-31a6e25a83d7" (UID: "d68bc927-978e-47f9-8510-31a6e25a83d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:29.018685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.018672 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68bc927-978e-47f9-8510-31a6e25a83d7-kube-api-access-gtknx" pod "d68bc927-978e-47f9-8510-31a6e25a83d7" (UID: "d68bc927-978e-47f9-8510-31a6e25a83d7"). InnerVolumeSpecName "kube-api-access-gtknx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:29.118372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.118337 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68bc927-978e-47f9-8510-31a6e25a83d7-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.118372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.118367 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtknx\" (UniqueName: \"kubernetes.io/projected/d68bc927-978e-47f9-8510-31a6e25a83d7-kube-api-access-gtknx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.715586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.715514 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:24:29.733095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.733059 5001 generic.go:356] "Generic (PLEG): container finished" podID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerID="e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd" exitCode=0 Oct 30 06:24:29.733609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.733121 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" Oct 30 06:24:29.733609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.733140 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" event={"ID":"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1","Type":"ContainerDied","Data":"e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd"} Oct 30 06:24:29.733609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.733202 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km" event={"ID":"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1","Type":"ContainerDied","Data":"45cdf9a4ebc81bc2d0a5f5af91a3e4a70e6a2ac9353f4b006916b36024ce03b2"} Oct 30 06:24:29.733609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.733230 5001 scope.go:122] "RemoveContainer" containerID="e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd" Oct 30 06:24:29.735063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.735043 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" Oct 30 06:24:29.735170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.735070 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg" event={"ID":"d68bc927-978e-47f9-8510-31a6e25a83d7","Type":"ContainerDied","Data":"b9f0d482917f1de7df861b51cb10ecbfb4e89d4d8de1ab2393b7237d47c5d6ae"} Oct 30 06:24:29.735170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.735115 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9f0d482917f1de7df861b51cb10ecbfb4e89d4d8de1ab2393b7237d47c5d6ae" Oct 30 06:24:29.745636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.745614 5001 scope.go:122] "RemoveContainer" containerID="52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9" Oct 30 06:24:29.756229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.756202 5001 scope.go:122] "RemoveContainer" containerID="e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd" Oct 30 06:24:29.756547 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:24:29.756527 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd\": container with ID starting with e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd not found: ID does not exist" containerID="e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd" Oct 30 06:24:29.756616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.756557 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd"} err="failed to get container status \"e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd\": rpc error: code = NotFound desc = could not find container \"e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd\": container with ID starting with e281d2c0b5371470b195f574622fcdaf37a55177ed3026a1641f4ef0c11851cd not found: ID does not exist" Oct 30 06:24:29.756616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.756576 5001 scope.go:122] "RemoveContainer" containerID="52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9" Oct 30 06:24:29.756861 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:24:29.756845 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9\": container with ID starting with 52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9 not found: ID does not exist" containerID="52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9" Oct 30 06:24:29.756916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.756864 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9"} err="failed to get container status \"52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9\": rpc error: code = NotFound desc = could not find container \"52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9\": container with ID starting with 52ed0f4d8f25c45c6d2d213e12bbf60c0a11e67ef3c15bacda5fdd33e6053df9 not found: ID does not exist" Oct 30 06:24:29.827131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827095 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-token\" (UniqueName: \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-token\") pod \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " Oct 30 06:24:29.827342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827184 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-certs\") pod \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " Oct 30 06:24:29.827342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827289 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-envoy\") pod \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " Oct 30 06:24:29.827468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827376 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-socket\") pod \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " Oct 30 06:24:29.827771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827516 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-certs" pod "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" (UID: "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:29.827771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827567 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-credential-socket\") pod \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " Oct 30 06:24:29.827771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827622 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-podinfo\") pod \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " Oct 30 06:24:29.827771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827710 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-kube-api-access-4cfcz\" (UniqueName: \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-kube-api-access-4cfcz\") pod \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " Oct 30 06:24:29.827771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827754 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-data\") pod \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " Oct 30 06:24:29.828107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827795 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-socket" pod "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" (UID: "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:29.828107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827814 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istiod-ca-cert\") pod \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\" (UID: \"3a61afcf-f45e-48f2-80b6-3880d8bd5ad1\") " Oct 30 06:24:29.828107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.827936 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-credential-socket" pod "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" (UID: "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:29.828341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.828282 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-data" pod "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" (UID: "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:29.828451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.828329 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.828451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.828383 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.828451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.828399 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.828735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.828711 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istiod-ca-cert" pod "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" (UID: "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:29.830449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.830420 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-envoy" pod "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" (UID: "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:29.830601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.830422 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-token" pod "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" (UID: "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:29.830795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.830776 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-podinfo" pod "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" (UID: "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:29.830795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.830780 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-kube-api-access-4cfcz" pod "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" (UID: "3a61afcf-f45e-48f2-80b6-3880d8bd5ad1"). InnerVolumeSpecName "kube-api-access-4cfcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:29.930382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.930288 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.930382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.930380 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.930607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.930397 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.930607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.930411 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cfcz\" (UniqueName: \"kubernetes.io/projected/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-kube-api-access-4cfcz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.930607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.930424 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.930607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:29.930437 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:30.079197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:30.079153 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km"] Oct 30 06:24:30.092860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:30.092830 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-4s6km"] Oct 30 06:24:30.118939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:30.118901 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" path="/var/lib/kubelet/pods/3a61afcf-f45e-48f2-80b6-3880d8bd5ad1/volumes" Oct 30 06:24:31.435036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435000 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh"] Oct 30 06:24:31.435598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435582 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" Oct 30 06:24:31.435646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435600 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" Oct 30 06:24:31.435646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435614 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d68bc927-978e-47f9-8510-31a6e25a83d7" containerName="revoke-keys" Oct 30 06:24:31.435646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435620 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68bc927-978e-47f9-8510-31a6e25a83d7" containerName="revoke-keys" Oct 30 06:24:31.435743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435648 5001 cpu_manager.go:414] "containerMap: removing container" podUID="37065385-a554-4c6f-89a7-416395f6c02b" containerName="revoke-keys" Oct 30 06:24:31.435743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435653 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37065385-a554-4c6f-89a7-416395f6c02b" containerName="revoke-keys" Oct 30 06:24:31.435743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435673 5001 cpu_manager.go:414] "containerMap: removing container" podUID="15722fe8-c60d-4e9d-b5c2-bbe3b7b49947" containerName="revoke-keys" Oct 30 06:24:31.435743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435678 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15722fe8-c60d-4e9d-b5c2-bbe3b7b49947" containerName="revoke-keys" Oct 30 06:24:31.435861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435758 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="37065385-a554-4c6f-89a7-416395f6c02b" containerName="revoke-keys" Oct 30 06:24:31.435861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435769 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" Oct 30 06:24:31.435861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435775 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="15722fe8-c60d-4e9d-b5c2-bbe3b7b49947" containerName="revoke-keys" Oct 30 06:24:31.435861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435782 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" Oct 30 06:24:31.435861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.435788 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68bc927-978e-47f9-8510-31a6e25a83d7" containerName="revoke-keys" Oct 30 06:24:31.440867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.440849 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:31.457377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.457352 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:24:31.469211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.469184 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:24:31.473563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.473529 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh"] Oct 30 06:24:31.524754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.524718 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:24:31.524974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.524832 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:24:31.549578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.549542 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c61850-bd7sh\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:31.549816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.549614 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2rj\" (UniqueName: \"kubernetes.io/projected/675dcd63-75cb-451b-999d-79b1ac5017b9-kube-api-access-hg2rj\") pod \"maas-api-revoke-sub-a6c61850-bd7sh\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:31.549816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.549697 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c61850-bd7sh\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:31.651212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.651175 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c61850-bd7sh\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:31.651430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.651283 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2rj\" (UniqueName: \"kubernetes.io/projected/675dcd63-75cb-451b-999d-79b1ac5017b9-kube-api-access-hg2rj\") pod \"maas-api-revoke-sub-a6c61850-bd7sh\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:31.651430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.651367 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c61850-bd7sh\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:31.652031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.652003 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c61850-bd7sh\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:31.670157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.670123 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c61850-bd7sh\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:31.676058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.676028 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2rj\" (UniqueName: \"kubernetes.io/projected/675dcd63-75cb-451b-999d-79b1ac5017b9-kube-api-access-hg2rj\") pod \"maas-api-revoke-sub-a6c61850-bd7sh\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:31.754814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:31.754721 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:32.083622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:32.083592 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh"] Oct 30 06:24:32.086501 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:32.086464 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675dcd63_75cb_451b_999d_79b1ac5017b9.slice/crio-24c133be6aa2cc672febf5f7ee82595e9d0e9fb569b0dbde7d712a86961a1e41 WatchSource:0}: Error finding container 24c133be6aa2cc672febf5f7ee82595e9d0e9fb569b0dbde7d712a86961a1e41: Status 404 returned error can't find the container with id 24c133be6aa2cc672febf5f7ee82595e9d0e9fb569b0dbde7d712a86961a1e41 Oct 30 06:24:32.766572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:32.766478 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" event={"ID":"675dcd63-75cb-451b-999d-79b1ac5017b9","Type":"ContainerStarted","Data":"4cf06900233868f5640019138ad91c604c13f8c132dbe0b6e68c6ea25f60632c"} Oct 30 06:24:32.766949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:32.766578 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" event={"ID":"675dcd63-75cb-451b-999d-79b1ac5017b9","Type":"ContainerStarted","Data":"24c133be6aa2cc672febf5f7ee82595e9d0e9fb569b0dbde7d712a86961a1e41"} Oct 30 06:24:32.806531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:32.806453 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" podStartSLOduration=1.806435588 podStartE2EDuration="1.806435588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:32.805352223 +0000 UTC m=+1886.801309233" watchObservedRunningTime="2026-10-30 06:24:32.806435588 +0000 UTC m=+1886.802392599" Oct 30 06:24:33.775900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:33.775866 5001 generic.go:356] "Generic (PLEG): container finished" podID="675dcd63-75cb-451b-999d-79b1ac5017b9" containerID="4cf06900233868f5640019138ad91c604c13f8c132dbe0b6e68c6ea25f60632c" exitCode=0 Oct 30 06:24:33.776376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:33.776021 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" event={"ID":"675dcd63-75cb-451b-999d-79b1ac5017b9","Type":"ContainerDied","Data":"4cf06900233868f5640019138ad91c604c13f8c132dbe0b6e68c6ea25f60632c"} Oct 30 06:24:34.932693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:34.932666 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:34.996142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:34.996104 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-openshift-ca\") pod \"675dcd63-75cb-451b-999d-79b1ac5017b9\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " Oct 30 06:24:34.996359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:34.996239 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-xks-ca\") pod \"675dcd63-75cb-451b-999d-79b1ac5017b9\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " Oct 30 06:24:34.996577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:34.996548 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-openshift-ca" pod "675dcd63-75cb-451b-999d-79b1ac5017b9" (UID: "675dcd63-75cb-451b-999d-79b1ac5017b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:34.996878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:34.996684 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675dcd63-75cb-451b-999d-79b1ac5017b9-kube-api-access-hg2rj\" (UniqueName: \"kubernetes.io/projected/675dcd63-75cb-451b-999d-79b1ac5017b9-kube-api-access-hg2rj\") pod \"675dcd63-75cb-451b-999d-79b1ac5017b9\" (UID: \"675dcd63-75cb-451b-999d-79b1ac5017b9\") " Oct 30 06:24:34.997454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:34.997256 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:34.999291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:34.999264 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-xks-ca" pod "675dcd63-75cb-451b-999d-79b1ac5017b9" (UID: "675dcd63-75cb-451b-999d-79b1ac5017b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:34.999766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:34.999742 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675dcd63-75cb-451b-999d-79b1ac5017b9-kube-api-access-hg2rj" pod "675dcd63-75cb-451b-999d-79b1ac5017b9" (UID: "675dcd63-75cb-451b-999d-79b1ac5017b9"). InnerVolumeSpecName "kube-api-access-hg2rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:35.098330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:35.098222 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675dcd63-75cb-451b-999d-79b1ac5017b9-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:35.098330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:35.098252 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg2rj\" (UniqueName: \"kubernetes.io/projected/675dcd63-75cb-451b-999d-79b1ac5017b9-kube-api-access-hg2rj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:35.792082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:35.792049 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" Oct 30 06:24:35.792082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:35.792063 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh" event={"ID":"675dcd63-75cb-451b-999d-79b1ac5017b9","Type":"ContainerDied","Data":"24c133be6aa2cc672febf5f7ee82595e9d0e9fb569b0dbde7d712a86961a1e41"} Oct 30 06:24:35.792404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:35.792108 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24c133be6aa2cc672febf5f7ee82595e9d0e9fb569b0dbde7d712a86961a1e41" Oct 30 06:24:37.348388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.348345 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l"] Oct 30 06:24:37.349191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.349167 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" Oct 30 06:24:37.349352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.349193 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a61afcf-f45e-48f2-80b6-3880d8bd5ad1" containerName="istio-proxy" Oct 30 06:24:37.349352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.349214 5001 cpu_manager.go:414] "containerMap: removing container" podUID="675dcd63-75cb-451b-999d-79b1ac5017b9" containerName="revoke-keys" Oct 30 06:24:37.349352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.349221 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675dcd63-75cb-451b-999d-79b1ac5017b9" containerName="revoke-keys" Oct 30 06:24:37.349525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.349384 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="675dcd63-75cb-451b-999d-79b1ac5017b9" containerName="revoke-keys" Oct 30 06:24:37.354636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.354607 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.375813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.375753 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz"] Oct 30 06:24:37.379875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.379847 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.382140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.382117 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-aaf467-openshift-default-dockercfg-txgqd\"" Oct 30 06:24:37.388376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.388351 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l"] Oct 30 06:24:37.412324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.412282 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz"] Oct 30 06:24:37.446147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446102 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.446295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446160 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f26970b-99c6-408d-be28-3a0938db99b6-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.446295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446275 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.446449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446374 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr7kn\" (UniqueName: \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-kube-api-access-pr7kn\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.446449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446402 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.446449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446430 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.446626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446532 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.446626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446609 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrcmx\" (UniqueName: \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-kube-api-access-zrcmx\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.446752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446646 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.446752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446693 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.446752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446733 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f26970b-99c6-408d-be28-3a0938db99b6-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.446940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446814 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.446940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446919 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.447051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.446953 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.447093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.447040 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.447137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.447089 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.447137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.447116 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.447215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.447194 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.500666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.500633 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml"] Oct 30 06:24:37.504799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.504783 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:37.521388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.521227 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:24:37.521388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.521250 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:24:37.521572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.521435 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:24:37.521659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.521633 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:24:37.525756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.525730 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml"] Oct 30 06:24:37.548740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.548708 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.548861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.548747 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr7kn\" (UniqueName: \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-kube-api-access-pr7kn\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.548861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.548767 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.548955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.548885 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.548955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.548939 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.549061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549017 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrcmx\" (UniqueName: \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-kube-api-access-zrcmx\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.549061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549053 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.549170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549087 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.549170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549111 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f26970b-99c6-408d-be28-3a0938db99b6-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.549170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549140 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.549434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549172 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.549434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549179 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.549434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549328 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.549604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549455 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.549604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549483 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.549604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549488 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.549604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549528 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.549604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549570 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.549865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549620 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.549865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549651 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.549865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549737 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.549865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549743 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.549865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549807 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.549865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549794 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f26970b-99c6-408d-be28-3a0938db99b6-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.550385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549920 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.550385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.549984 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.550385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.550271 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f26970b-99c6-408d-be28-3a0938db99b6-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.550776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.550751 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.551922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.551893 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.552107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.552083 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f26970b-99c6-408d-be28-3a0938db99b6-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.552590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.552571 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.553869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.553849 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.571651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.571621 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.575606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.575584 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrcmx\" (UniqueName: \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-kube-api-access-zrcmx\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.576484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.576465 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.577098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.577079 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr7kn\" (UniqueName: \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-kube-api-access-pr7kn\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.651730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.651700 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b59102a7-8mxml\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:37.651891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.651738 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf8h4\" (UniqueName: \"kubernetes.io/projected/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-kube-api-access-nf8h4\") pod \"maas-api-revoke-sub-b59102a7-8mxml\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:37.651953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.651900 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b59102a7-8mxml\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:37.676883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.676860 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:37.697246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.697218 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:37.755623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.753633 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b59102a7-8mxml\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:37.755623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.753794 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b59102a7-8mxml\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:37.755623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.753822 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf8h4\" (UniqueName: \"kubernetes.io/projected/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-kube-api-access-nf8h4\") pod \"maas-api-revoke-sub-b59102a7-8mxml\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:37.755623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.754737 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b59102a7-8mxml\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:37.765523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.765493 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b59102a7-8mxml\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:37.784023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.783441 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf8h4\" (UniqueName: \"kubernetes.io/projected/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-kube-api-access-nf8h4\") pod \"maas-api-revoke-sub-b59102a7-8mxml\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:37.819428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:37.819394 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:38.008346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.008317 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l"] Oct 30 06:24:38.010127 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:38.010098 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3a6b7c_98e8_4192_aad0_1b917966ffed.slice/crio-999b5f80c909ffaf706a6ffb5f54bdfd526569582ecba3a277968e0f9b734de5 WatchSource:0}: Error finding container 999b5f80c909ffaf706a6ffb5f54bdfd526569582ecba3a277968e0f9b734de5: Status 404 returned error can't find the container with id 999b5f80c909ffaf706a6ffb5f54bdfd526569582ecba3a277968e0f9b734de5 Oct 30 06:24:38.012795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.012759 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:38.012893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.012842 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:38.012893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.012887 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:38.025856 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:38.025830 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f26970b_99c6_408d_be28_3a0938db99b6.slice/crio-a1b4ba6cf0b74898614a1cd39ae9600828c550f2281b6772af637b9409d2b958 WatchSource:0}: Error finding container a1b4ba6cf0b74898614a1cd39ae9600828c550f2281b6772af637b9409d2b958: Status 404 returned error can't find the container with id a1b4ba6cf0b74898614a1cd39ae9600828c550f2281b6772af637b9409d2b958 Oct 30 06:24:38.026015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.025985 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz"] Oct 30 06:24:38.028566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.028533 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:38.028644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.028612 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:38.028693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.028649 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:38.144350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.143758 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml"] Oct 30 06:24:38.377179 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:24:38.377138 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:24:38.940332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.939291 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" event={"ID":"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e","Type":"ContainerStarted","Data":"1b2d483c08b28b0292337975c816af6b9c3d93b2df67f7cd58fa9b5c50a75178"} Oct 30 06:24:38.940332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.939413 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" event={"ID":"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e","Type":"ContainerStarted","Data":"e2999e43e60f4e78c431834a4e752196be22603307e3b5238de2c999ac3ba92a"} Oct 30 06:24:38.944330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.942255 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" event={"ID":"5c3a6b7c-98e8-4192-aad0-1b917966ffed","Type":"ContainerStarted","Data":"9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9"} Oct 30 06:24:38.944330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.942346 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" event={"ID":"5c3a6b7c-98e8-4192-aad0-1b917966ffed","Type":"ContainerStarted","Data":"999b5f80c909ffaf706a6ffb5f54bdfd526569582ecba3a277968e0f9b734de5"} Oct 30 06:24:38.966574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.961478 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" event={"ID":"1f26970b-99c6-408d-be28-3a0938db99b6","Type":"ContainerStarted","Data":"b7012eaaf3159d9a8fb6fc5a63d08ed8891ea16c729df1794612ac295c1957ca"} Oct 30 06:24:38.966574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:38.961555 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" event={"ID":"1f26970b-99c6-408d-be28-3a0938db99b6","Type":"ContainerStarted","Data":"a1b4ba6cf0b74898614a1cd39ae9600828c550f2281b6772af637b9409d2b958"} Oct 30 06:24:39.010264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.010203 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" podStartSLOduration=2.010188073 podStartE2EDuration="2.010188073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:38.983620585 +0000 UTC m=+1892.979577596" watchObservedRunningTime="2026-10-30 06:24:39.010188073 +0000 UTC m=+1893.006145077" Oct 30 06:24:39.040986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.040935 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" podStartSLOduration=2.040921167 podStartE2EDuration="2.040921167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:39.040059679 +0000 UTC m=+1893.036016691" watchObservedRunningTime="2026-10-30 06:24:39.040921167 +0000 UTC m=+1893.036878177" Oct 30 06:24:39.682331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.681694 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:39.682331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.681823 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:24:39.682331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.681870 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" podUID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 06:24:39.707333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.701112 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:39.707333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.705671 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 06:24:39.707333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.705728 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" podUID="1f26970b-99c6-408d-be28-3a0938db99b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 06:24:39.841422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.841363 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" podStartSLOduration=2.841346501 podStartE2EDuration="2.841346501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:39.110108177 +0000 UTC m=+1893.106065190" watchObservedRunningTime="2026-10-30 06:24:39.841346501 +0000 UTC m=+1893.837303565" Oct 30 06:24:39.843289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.843266 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8"] Oct 30 06:24:39.847628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.847487 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:39.870294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.866690 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8d44dd-openshift-default-dockercfg-nzqwp\"" Oct 30 06:24:39.899559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:39.899337 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8"] Oct 30 06:24:40.031038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.030509 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7kxx\" (UniqueName: \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-kube-api-access-x7kxx\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.031038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.030563 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.031038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.030590 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.031038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.030648 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.031038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.030719 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.031038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.030815 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9396bf9-d598-4311-84fa-28f84fa4c341-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.031038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.030847 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.031038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.030912 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.031038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.030954 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.031961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.031934 5001 generic.go:356] "Generic (PLEG): container finished" podID="ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e" containerID="1b2d483c08b28b0292337975c816af6b9c3d93b2df67f7cd58fa9b5c50a75178" exitCode=0 Oct 30 06:24:40.032428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.032401 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" event={"ID":"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e","Type":"ContainerDied","Data":"1b2d483c08b28b0292337975c816af6b9c3d93b2df67f7cd58fa9b5c50a75178"} Oct 30 06:24:40.132367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.132328 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.132551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.132398 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.132551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.132479 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7kxx\" (UniqueName: \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-kube-api-access-x7kxx\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.132551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.132526 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.132743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.132550 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.132743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.132647 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.132743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.132695 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.132907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.132796 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9396bf9-d598-4311-84fa-28f84fa4c341-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.132907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.132809 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.132907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.132848 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.132907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.132900 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.133150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.133121 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.133607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.133584 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.133922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.133897 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9396bf9-d598-4311-84fa-28f84fa4c341-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.135427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.135394 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.135586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.135550 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.213339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.209928 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7kxx\" (UniqueName: \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-kube-api-access-x7kxx\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.218351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.216112 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.493330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.493221 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:40.759383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.711493 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 06:24:40.759383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.711555 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" podUID="1f26970b-99c6-408d-be28-3a0938db99b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 06:24:40.759383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.712452 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:24:40.759383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:40.712506 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" podUID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 06:24:41.123859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.123814 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8"] Oct 30 06:24:41.129647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.129614 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:41.129741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.129678 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:41.129741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.129709 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:41.461335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.459897 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:41.503271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.491148 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-openshift-ca\") pod \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " Oct 30 06:24:41.503271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.491397 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-kube-api-access-nf8h4\" (UniqueName: \"kubernetes.io/projected/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-kube-api-access-nf8h4\") pod \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " Oct 30 06:24:41.503271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.491441 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-xks-ca\") pod \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\" (UID: \"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e\") " Oct 30 06:24:41.503271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.496494 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-openshift-ca" pod "ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e" (UID: "ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:41.533560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.533503 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-kube-api-access-nf8h4" pod "ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e" (UID: "ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e"). InnerVolumeSpecName "kube-api-access-nf8h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:41.534163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.534141 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-xks-ca" pod "ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e" (UID: "ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:41.610373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.593545 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:41.610373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.593585 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf8h4\" (UniqueName: \"kubernetes.io/projected/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-kube-api-access-nf8h4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:41.610373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.593601 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:41.715345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.684336 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:24:41.715345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.684398 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" podUID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 06:24:41.715345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.711516 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 06:24:41.715345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:41.711567 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" podUID="1f26970b-99c6-408d-be28-3a0938db99b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 06:24:42.050780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.050694 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" event={"ID":"b9396bf9-d598-4311-84fa-28f84fa4c341","Type":"ContainerStarted","Data":"c65c857a335be8d62275e352a1289212bae86a62fc74d80783b756180ec652fa"} Oct 30 06:24:42.050780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.050748 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" event={"ID":"b9396bf9-d598-4311-84fa-28f84fa4c341","Type":"ContainerStarted","Data":"5c6cfa646b6bb9b826c7460b80236ea24d986cb57987fcb585e7918b5fd2bda8"} Oct 30 06:24:42.072331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.070848 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" event={"ID":"ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e","Type":"ContainerDied","Data":"e2999e43e60f4e78c431834a4e752196be22603307e3b5238de2c999ac3ba92a"} Oct 30 06:24:42.072331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.070914 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2999e43e60f4e78c431834a4e752196be22603307e3b5238de2c999ac3ba92a" Oct 30 06:24:42.072331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.070987 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml" Oct 30 06:24:42.123448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.123395 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" podStartSLOduration=3.123381869 podStartE2EDuration="3.123381869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:42.122945469 +0000 UTC m=+1896.118902480" watchObservedRunningTime="2026-10-30 06:24:42.123381869 +0000 UTC m=+1896.119338879" Oct 30 06:24:42.519657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.494702 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:42.544587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.533472 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 06:24:42.544587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.533537 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 06:24:42.689333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.681236 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:24:42.689333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.681293 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" podUID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 06:24:42.704966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.704924 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 06:24:42.709427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:42.709382 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" podUID="1f26970b-99c6-408d-be28-3a0938db99b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 06:24:43.496331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:43.494720 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 06:24:43.496331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:43.494796 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 06:24:43.678170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:43.677734 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:24:43.678170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:43.677791 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" podUID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 06:24:43.701519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:43.701477 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 06:24:43.701761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:43.701735 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" podUID="1f26970b-99c6-408d-be28-3a0938db99b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 06:24:44.493728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:44.493681 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 06:24:44.494031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:44.494002 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 06:24:44.687331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:44.686734 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:24:44.687331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:44.686800 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" podUID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 06:24:44.708328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:44.698993 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 06:24:44.708328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:44.699052 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" podUID="1f26970b-99c6-408d-be28-3a0938db99b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 06:24:45.495737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:45.495704 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 06:24:45.495977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:45.495947 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 06:24:46.395640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.395607 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:46.397605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.397583 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:46.402552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.402528 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:24:46.462346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.459632 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:46.462346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.461567 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:46.471330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.468542 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:24:46.497327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.493859 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 06:24:46.497327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.493910 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 06:24:46.818381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.818281 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p"] Oct 30 06:24:46.818932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.818916 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e" containerName="revoke-keys" Oct 30 06:24:46.818932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.818932 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e" containerName="revoke-keys" Oct 30 06:24:46.819041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.819032 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e" containerName="revoke-keys" Oct 30 06:24:46.822551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.822534 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:46.836434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.836412 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6drxn\" (UniqueName: \"kubernetes.io/projected/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-kube-api-access-6drxn\") pod \"maas-api-revoke-sub-9b7a71b7-rjx7p\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:46.836550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.836530 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b7a71b7-rjx7p\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:46.836634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.836621 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b7a71b7-rjx7p\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:46.837937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.837775 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:24:46.838101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.837939 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:24:46.838248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.838102 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:24:46.838248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.838005 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:24:46.842533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.842511 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p"] Oct 30 06:24:46.937454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.937427 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b7a71b7-rjx7p\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:46.937642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.937516 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6drxn\" (UniqueName: \"kubernetes.io/projected/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-kube-api-access-6drxn\") pod \"maas-api-revoke-sub-9b7a71b7-rjx7p\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:46.937642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.937583 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b7a71b7-rjx7p\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:46.938063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.938037 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b7a71b7-rjx7p\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:46.940017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.939996 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b7a71b7-rjx7p\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:46.980368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:46.980329 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6drxn\" (UniqueName: \"kubernetes.io/projected/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-kube-api-access-6drxn\") pod \"maas-api-revoke-sub-9b7a71b7-rjx7p\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:47.136460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:47.136430 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:47.486581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:47.486554 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p"] Oct 30 06:24:47.487829 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:47.487803 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb586cb3_d7bf_4ad7_9249_8555759a7bd0.slice/crio-e2cdab8099f5ced763f6aadff4788d85c7bde5d4fce55f3f3e8e0f04f058d0eb WatchSource:0}: Error finding container e2cdab8099f5ced763f6aadff4788d85c7bde5d4fce55f3f3e8e0f04f058d0eb: Status 404 returned error can't find the container with id e2cdab8099f5ced763f6aadff4788d85c7bde5d4fce55f3f3e8e0f04f058d0eb Oct 30 06:24:47.493863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:47.493839 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 06:24:47.493984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:47.493891 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 06:24:48.222816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:48.222718 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" event={"ID":"eb586cb3-d7bf-4ad7-9249-8555759a7bd0","Type":"ContainerStarted","Data":"e1259295f3855d23e0ae090cda4cf146b64e1ad1bc0ee87160c17684e86294dc"} Oct 30 06:24:48.222994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:48.222825 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" event={"ID":"eb586cb3-d7bf-4ad7-9249-8555759a7bd0","Type":"ContainerStarted","Data":"e2cdab8099f5ced763f6aadff4788d85c7bde5d4fce55f3f3e8e0f04f058d0eb"} Oct 30 06:24:48.264084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:48.264026 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" podStartSLOduration=2.264006696 podStartE2EDuration="2.264006696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:48.260794978 +0000 UTC m=+1902.256751990" watchObservedRunningTime="2026-10-30 06:24:48.264006696 +0000 UTC m=+1902.259963711" Oct 30 06:24:48.498515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:48.496809 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 06:24:48.498515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:48.496888 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 06:24:48.899328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:48.898956 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn"] Oct 30 06:24:48.911146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:48.911094 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f2cf8a-nfjpn"] Oct 30 06:24:49.239721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:49.239617 5001 generic.go:356] "Generic (PLEG): container finished" podID="eb586cb3-d7bf-4ad7-9249-8555759a7bd0" containerID="e1259295f3855d23e0ae090cda4cf146b64e1ad1bc0ee87160c17684e86294dc" exitCode=0 Oct 30 06:24:49.239943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:49.239745 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" event={"ID":"eb586cb3-d7bf-4ad7-9249-8555759a7bd0","Type":"ContainerDied","Data":"e1259295f3855d23e0ae090cda4cf146b64e1ad1bc0ee87160c17684e86294dc"} Oct 30 06:24:50.138139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.138097 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37065385-a554-4c6f-89a7-416395f6c02b" path="/var/lib/kubelet/pods/37065385-a554-4c6f-89a7-416395f6c02b/volumes" Oct 30 06:24:50.505330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.505227 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:24:50.505330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.505288 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:24:50.609227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.609199 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:50.641648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.641616 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj"] Oct 30 06:24:50.642681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.642657 5001 cpu_manager.go:414] "containerMap: removing container" podUID="eb586cb3-d7bf-4ad7-9249-8555759a7bd0" containerName="revoke-keys" Oct 30 06:24:50.642831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.642683 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb586cb3-d7bf-4ad7-9249-8555759a7bd0" containerName="revoke-keys" Oct 30 06:24:50.642889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.642872 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb586cb3-d7bf-4ad7-9249-8555759a7bd0" containerName="revoke-keys" Oct 30 06:24:50.647018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.646987 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:50.647829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.647802 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-xks-ca\") pod \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " Oct 30 06:24:50.647964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.647923 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-kube-api-access-6drxn\" (UniqueName: \"kubernetes.io/projected/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-kube-api-access-6drxn\") pod \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " Oct 30 06:24:50.648359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.648033 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-openshift-ca\") pod \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\" (UID: \"eb586cb3-d7bf-4ad7-9249-8555759a7bd0\") " Oct 30 06:24:50.648983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.648956 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-openshift-ca" pod "eb586cb3-d7bf-4ad7-9249-8555759a7bd0" (UID: "eb586cb3-d7bf-4ad7-9249-8555759a7bd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:50.673839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.664782 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-kube-api-access-6drxn" pod "eb586cb3-d7bf-4ad7-9249-8555759a7bd0" (UID: "eb586cb3-d7bf-4ad7-9249-8555759a7bd0"). InnerVolumeSpecName "kube-api-access-6drxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:50.673839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.666536 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-xks-ca" pod "eb586cb3-d7bf-4ad7-9249-8555759a7bd0" (UID: "eb586cb3-d7bf-4ad7-9249-8555759a7bd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:50.697937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.697852 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj"] Oct 30 06:24:50.749992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.749943 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d43f67bf-bhjtj\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:50.750409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.750387 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gdj\" (UniqueName: \"kubernetes.io/projected/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-kube-api-access-j4gdj\") pod \"maas-api-revoke-sub-d43f67bf-bhjtj\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:50.750688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.750662 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d43f67bf-bhjtj\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:50.750793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.750759 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6drxn\" (UniqueName: \"kubernetes.io/projected/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-kube-api-access-6drxn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:50.750793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.750778 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:50.750899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.750795 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb586cb3-d7bf-4ad7-9249-8555759a7bd0-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:50.852184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.852153 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d43f67bf-bhjtj\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:50.852408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.852245 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d43f67bf-bhjtj\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:50.852408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.852380 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gdj\" (UniqueName: \"kubernetes.io/projected/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-kube-api-access-j4gdj\") pod \"maas-api-revoke-sub-d43f67bf-bhjtj\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:50.853733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.853636 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d43f67bf-bhjtj\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:50.855562 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.855537 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d43f67bf-bhjtj\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:50.875713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:50.875689 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gdj\" (UniqueName: \"kubernetes.io/projected/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-kube-api-access-j4gdj\") pod \"maas-api-revoke-sub-d43f67bf-bhjtj\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:51.017625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.017542 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:51.136797 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.134795 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7"] Oct 30 06:24:51.174425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.172545 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7"] Oct 30 06:24:51.174425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.172932 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:51.286117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.286024 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:51.286591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.286564 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:51.286905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.286884 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btp67\" (UniqueName: \"kubernetes.io/projected/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-kube-api-access-btp67\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:51.323445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.323412 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" event={"ID":"eb586cb3-d7bf-4ad7-9249-8555759a7bd0","Type":"ContainerDied","Data":"e2cdab8099f5ced763f6aadff4788d85c7bde5d4fce55f3f3e8e0f04f058d0eb"} Oct 30 06:24:51.323589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.323472 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2cdab8099f5ced763f6aadff4788d85c7bde5d4fce55f3f3e8e0f04f058d0eb" Oct 30 06:24:51.323589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.323432 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p" Oct 30 06:24:51.388789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.388755 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:51.388982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.388863 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btp67\" (UniqueName: \"kubernetes.io/projected/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-kube-api-access-btp67\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:51.388982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.388932 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:51.389736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.389710 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:51.392475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.392452 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:51.411971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.411945 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btp67\" (UniqueName: \"kubernetes.io/projected/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-kube-api-access-btp67\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:51.449666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.449642 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj"] Oct 30 06:24:51.454343 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:51.452779 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ebe4b0_8e32_4173_b3ea_141ab9e3fd8c.slice/crio-a71c80c46605d554c768895d13af32ced03507b0ca267dbad2636c5abc93c65b WatchSource:0}: Error finding container a71c80c46605d554c768895d13af32ced03507b0ca267dbad2636c5abc93c65b: Status 404 returned error can't find the container with id a71c80c46605d554c768895d13af32ced03507b0ca267dbad2636c5abc93c65b Oct 30 06:24:51.497867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.497840 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:51.513982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.513948 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:24:51.514097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.514014 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:24:51.642608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.642558 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:51.643878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.643850 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:51.645747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.645605 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:24:51.826236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:51.826200 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7"] Oct 30 06:24:51.828144 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:51.828120 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aec366f_9c24_4dc7_b1bf_de7ebeefcb8b.slice/crio-4d4922461df7c146c49f50a30685a9325f9f6d1c5908e12bb6a96538fd9a1aaa WatchSource:0}: Error finding container 4d4922461df7c146c49f50a30685a9325f9f6d1c5908e12bb6a96538fd9a1aaa: Status 404 returned error can't find the container with id 4d4922461df7c146c49f50a30685a9325f9f6d1c5908e12bb6a96538fd9a1aaa Oct 30 06:24:52.332524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.332484 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" event={"ID":"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b","Type":"ContainerStarted","Data":"936f936bd93019790b694c558ec63e204060fbd18d259b9fd38a0a302a88b451"} Oct 30 06:24:52.333262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.332546 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" event={"ID":"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b","Type":"ContainerStarted","Data":"4d4922461df7c146c49f50a30685a9325f9f6d1c5908e12bb6a96538fd9a1aaa"} Oct 30 06:24:52.334274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.334242 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" event={"ID":"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c","Type":"ContainerStarted","Data":"355c57cad80142931844c1dec499a106f34f32f33c8d7f07ee0ed5edff0b1d58"} Oct 30 06:24:52.334454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.334297 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" event={"ID":"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c","Type":"ContainerStarted","Data":"a71c80c46605d554c768895d13af32ced03507b0ca267dbad2636c5abc93c65b"} Oct 30 06:24:52.379689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.379622 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" podStartSLOduration=1.379603749 podStartE2EDuration="1.379603749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:52.373609249 +0000 UTC m=+1906.369566263" watchObservedRunningTime="2026-10-30 06:24:52.379603749 +0000 UTC m=+1906.375560763" Oct 30 06:24:52.410334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.408154 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26"] Oct 30 06:24:52.419696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.419654 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.420321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.420265 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" podStartSLOduration=2.420253296 podStartE2EDuration="2.420253296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:52.418232386 +0000 UTC m=+1906.414189396" watchObservedRunningTime="2026-10-30 06:24:52.420253296 +0000 UTC m=+1906.416210338" Oct 30 06:24:52.435188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.435160 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26"] Oct 30 06:24:52.509557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.509522 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.509683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.509586 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.509683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.509669 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.509751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.509729 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.509886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.509864 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhx6\" (UniqueName: \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-kube-api-access-vbhx6\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.509921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.509903 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.509953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.509922 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.509985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.509968 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.510149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.510137 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.610915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.610881 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.610923 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.610978 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.611033 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhx6\" (UniqueName: \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-kube-api-access-vbhx6\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.611065 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.611087 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.611108 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.611240 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.611343 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.611463 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611655 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.611617 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611655 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.611640 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.611718 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.611778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.611752 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.613635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.613617 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.613785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.613764 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.638511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.638479 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.670470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.669255 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhx6\" (UniqueName: \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-kube-api-access-vbhx6\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:52.737875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:52.737799 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:53.068464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.068435 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26"] Oct 30 06:24:53.071011 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:53.070978 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5d02f0_ca64_4cbf_be9b_7003996c44d8.slice/crio-9cb08173803ff048bb261d748c70875ffcd091c48758cbf518e39b5fc73148a0 WatchSource:0}: Error finding container 9cb08173803ff048bb261d748c70875ffcd091c48758cbf518e39b5fc73148a0: Status 404 returned error can't find the container with id 9cb08173803ff048bb261d748c70875ffcd091c48758cbf518e39b5fc73148a0 Oct 30 06:24:53.074743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.074170 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:53.074743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.074270 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:53.074743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.074358 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:53.352348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.352240 5001 generic.go:356] "Generic (PLEG): container finished" podID="6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b" containerID="936f936bd93019790b694c558ec63e204060fbd18d259b9fd38a0a302a88b451" exitCode=0 Oct 30 06:24:53.352844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.352437 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" event={"ID":"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b","Type":"ContainerDied","Data":"936f936bd93019790b694c558ec63e204060fbd18d259b9fd38a0a302a88b451"} Oct 30 06:24:53.355048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.355021 5001 generic.go:356] "Generic (PLEG): container finished" podID="73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c" containerID="355c57cad80142931844c1dec499a106f34f32f33c8d7f07ee0ed5edff0b1d58" exitCode=0 Oct 30 06:24:53.355172 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.355141 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" event={"ID":"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c","Type":"ContainerDied","Data":"355c57cad80142931844c1dec499a106f34f32f33c8d7f07ee0ed5edff0b1d58"} Oct 30 06:24:53.358867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.358847 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" event={"ID":"0d5d02f0-ca64-4cbf-be9b-7003996c44d8","Type":"ContainerStarted","Data":"a80c2cacb8f75b5fce2b69b90943b8a309448ba2aa3319ceda0dfd41e075c37b"} Oct 30 06:24:53.359037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.359021 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" event={"ID":"0d5d02f0-ca64-4cbf-be9b-7003996c44d8","Type":"ContainerStarted","Data":"9cb08173803ff048bb261d748c70875ffcd091c48758cbf518e39b5fc73148a0"} Oct 30 06:24:53.416504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.416478 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg"] Oct 30 06:24:53.419997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.419970 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a035a3-8tgvg"] Oct 30 06:24:53.500186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.500102 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" podStartSLOduration=1.50008293 podStartE2EDuration="1.50008293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:53.498200396 +0000 UTC m=+1907.494157442" watchObservedRunningTime="2026-10-30 06:24:53.50008293 +0000 UTC m=+1907.496039943" Oct 30 06:24:53.738087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:53.738058 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:24:54.117559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.117522 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68bc927-978e-47f9-8510-31a6e25a83d7" path="/var/lib/kubelet/pods/d68bc927-978e-47f9-8510-31a6e25a83d7/volumes" Oct 30 06:24:54.638605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.638582 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:54.707366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.707075 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:54.739676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.739633 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:24:54.739867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.739700 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" podUID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 06:24:54.795107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.795047 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-kube-api-access-btp67\" (UniqueName: \"kubernetes.io/projected/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-kube-api-access-btp67\") pod \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " Oct 30 06:24:54.795466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.795417 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-xks-ca\") pod \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " Oct 30 06:24:54.795838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.795801 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-openshift-ca\") pod \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " Oct 30 06:24:54.796099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.796080 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-openshift-ca\") pod \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " Oct 30 06:24:54.796716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.796685 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-xks-ca\") pod \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\" (UID: \"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b\") " Oct 30 06:24:54.797281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.797239 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-kube-api-access-j4gdj\" (UniqueName: \"kubernetes.io/projected/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-kube-api-access-j4gdj\") pod \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\" (UID: \"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c\") " Oct 30 06:24:54.797682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.796604 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-openshift-ca" pod "6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b" (UID: "6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:54.797821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.796605 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-openshift-ca" pod "73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c" (UID: "73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:54.798217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.798190 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-xks-ca" pod "73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c" (UID: "73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:54.798821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.798803 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:54.798974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.798958 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:54.799107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.799095 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:54.801558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.801532 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-xks-ca" pod "6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b" (UID: "6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:54.802757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.802733 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-kube-api-access-btp67" pod "6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b" (UID: "6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b"). InnerVolumeSpecName "kube-api-access-btp67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:54.803079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.803058 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-kube-api-access-j4gdj" pod "73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c" (UID: "73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c"). InnerVolumeSpecName "kube-api-access-j4gdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:54.900872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.900787 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:54.900872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.900819 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4gdj\" (UniqueName: \"kubernetes.io/projected/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c-kube-api-access-j4gdj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:54.900872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:54.900830 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btp67\" (UniqueName: \"kubernetes.io/projected/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b-kube-api-access-btp67\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:24:55.404236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:55.401947 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" Oct 30 06:24:55.404236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:55.401983 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7" event={"ID":"6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b","Type":"ContainerDied","Data":"4d4922461df7c146c49f50a30685a9325f9f6d1c5908e12bb6a96538fd9a1aaa"} Oct 30 06:24:55.404236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:55.402043 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d4922461df7c146c49f50a30685a9325f9f6d1c5908e12bb6a96538fd9a1aaa" Oct 30 06:24:55.405825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:55.405806 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" Oct 30 06:24:55.406012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:55.405825 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj" event={"ID":"73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c","Type":"ContainerDied","Data":"a71c80c46605d554c768895d13af32ced03507b0ca267dbad2636c5abc93c65b"} Oct 30 06:24:55.406135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:55.406124 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a71c80c46605d554c768895d13af32ced03507b0ca267dbad2636c5abc93c65b" Oct 30 06:24:55.738570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:55.738475 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:24:55.738570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:55.738530 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" podUID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 06:24:56.740331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:56.740019 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:24:56.740331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:56.740080 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" podUID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 06:24:56.940686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:56.940651 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5"] Oct 30 06:24:56.941191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:56.940860 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" containerID="cri-o://03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d" gracePeriod=30 Oct 30 06:24:57.374336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.372890 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz"] Oct 30 06:24:57.374336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.373827 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b" containerName="revoke-keys" Oct 30 06:24:57.374336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.373847 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b" containerName="revoke-keys" Oct 30 06:24:57.374336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.373901 5001 cpu_manager.go:414] "containerMap: removing container" podUID="73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c" containerName="revoke-keys" Oct 30 06:24:57.374336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.373911 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c" containerName="revoke-keys" Oct 30 06:24:57.374336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.374058 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b" containerName="revoke-keys" Oct 30 06:24:57.374336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.374077 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c" containerName="revoke-keys" Oct 30 06:24:57.382087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.382062 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:24:57.383943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.383904 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz"] Oct 30 06:24:57.399772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.399748 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:24:57.400455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.400435 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:24:57.400599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.400568 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:24:57.400669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.400618 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:24:57.484175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.484135 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8f26ca3-wkhjz\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:24:57.484375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.484202 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8f26ca3-wkhjz\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:24:57.484375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.484281 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dm57\" (UniqueName: \"kubernetes.io/projected/071940ca-738b-477f-a37b-6f179bc82f8b-kube-api-access-2dm57\") pod \"maas-api-revoke-sub-f8f26ca3-wkhjz\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:24:57.588328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.587290 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8f26ca3-wkhjz\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:24:57.588328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.587380 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dm57\" (UniqueName: \"kubernetes.io/projected/071940ca-738b-477f-a37b-6f179bc82f8b-kube-api-access-2dm57\") pod \"maas-api-revoke-sub-f8f26ca3-wkhjz\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:24:57.588328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.587710 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8f26ca3-wkhjz\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:24:57.594330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.588784 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8f26ca3-wkhjz\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:24:57.597815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.597738 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8f26ca3-wkhjz\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:24:57.629175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.629091 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dm57\" (UniqueName: \"kubernetes.io/projected/071940ca-738b-477f-a37b-6f179bc82f8b-kube-api-access-2dm57\") pod \"maas-api-revoke-sub-f8f26ca3-wkhjz\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:24:57.700541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.700506 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:24:57.738790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.738756 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:24:57.739093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:57.739043 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" podUID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 06:24:58.047067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:58.047042 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz"] Oct 30 06:24:58.049751 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:58.049719 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071940ca_738b_477f_a37b_6f179bc82f8b.slice/crio-36bb794ac026b2c4d5f64b371e82070b90badec0ed4f2252e16b27f6929b705f WatchSource:0}: Error finding container 36bb794ac026b2c4d5f64b371e82070b90badec0ed4f2252e16b27f6929b705f: Status 404 returned error can't find the container with id 36bb794ac026b2c4d5f64b371e82070b90badec0ed4f2252e16b27f6929b705f Oct 30 06:24:58.378464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:58.378376 5001 patch_prober.go:29] interesting pod/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:24:58.378662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:58.378458 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:24:58.433472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:58.433433 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" event={"ID":"071940ca-738b-477f-a37b-6f179bc82f8b","Type":"ContainerStarted","Data":"36bb794ac026b2c4d5f64b371e82070b90badec0ed4f2252e16b27f6929b705f"} Oct 30 06:24:58.738680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:58.738590 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:24:58.738680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:58.738649 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" podUID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 06:24:59.120741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.120698 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z"] Oct 30 06:24:59.125702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.125682 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.144796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.140908 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f5e55f-openshift-default-dockercfg-27csz\"" Oct 30 06:24:59.195985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.195922 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z"] Oct 30 06:24:59.217848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.217807 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-envoy\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.218020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.217867 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istiod-ca-cert\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.218020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.217997 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-socket\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.218147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.218026 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh888\" (UniqueName: \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-kube-api-access-nh888\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.218147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.218123 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-data\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.218263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.218188 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-certs\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.218263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.218223 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-podinfo\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.218400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.218344 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-token\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.218464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.218421 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-credential-socket\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.341549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.341447 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-data\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.352845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.352791 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-certs\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.361498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.361452 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-podinfo\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.385413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.361838 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-token\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.408296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.407018 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-credential-socket\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.408296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.407118 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-envoy\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.408296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.407174 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istiod-ca-cert\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.408296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.407203 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-socket\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.408296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.407229 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh888\" (UniqueName: \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-kube-api-access-nh888\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.408296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.407643 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-certs\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.408296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.407886 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-data\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.408296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.407994 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-credential-socket\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.445486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.411026 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istiod-ca-cert\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.445486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.426672 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-socket\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.482370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.473139 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh888\" (UniqueName: \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-kube-api-access-nh888\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.499839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.494922 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-token\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.500075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.499620 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-envoy\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.502838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.502815 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-podinfo\") pod \"e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.509465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.509440 5001 generic.go:356] "Generic (PLEG): container finished" podID="071940ca-738b-477f-a37b-6f179bc82f8b" containerID="9b3931b2ceb53a629bf1d7a143a7a5f1c060c6fa9d724422636355e49747f79a" exitCode=0 Oct 30 06:24:59.509585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.509522 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" event={"ID":"071940ca-738b-477f-a37b-6f179bc82f8b","Type":"ContainerDied","Data":"9b3931b2ceb53a629bf1d7a143a7a5f1c060c6fa9d724422636355e49747f79a"} Oct 30 06:24:59.514636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.514619 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:24:59.738682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.738646 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:24:59.738868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.738724 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" podUID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 06:24:59.960650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.960614 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z"] Oct 30 06:24:59.964064 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:24:59.964029 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ed5ca9_d682_4636_a96b_e7d2b84b8455.slice/crio-9a010b9c52a6721ab862f362862cd92ccdf2e5d668c80f959dbdb071428d5bfa WatchSource:0}: Error finding container 9a010b9c52a6721ab862f362862cd92ccdf2e5d668c80f959dbdb071428d5bfa: Status 404 returned error can't find the container with id 9a010b9c52a6721ab862f362862cd92ccdf2e5d668c80f959dbdb071428d5bfa Oct 30 06:24:59.967504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.967465 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:59.967597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.967564 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:59.967645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:24:59.967606 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:00.521885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:00.521852 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" event={"ID":"03ed5ca9-d682-4636-a96b-e7d2b84b8455","Type":"ContainerStarted","Data":"e1d702587347d396cebe122245345bb5a55542840cc54958553a06665dff9941"} Oct 30 06:25:00.522430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:00.521913 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" event={"ID":"03ed5ca9-d682-4636-a96b-e7d2b84b8455","Type":"ContainerStarted","Data":"9a010b9c52a6721ab862f362862cd92ccdf2e5d668c80f959dbdb071428d5bfa"} Oct 30 06:25:00.566920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:00.566856 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" podStartSLOduration=1.566836829 podStartE2EDuration="1.566836829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:00.566202279 +0000 UTC m=+1914.562159289" watchObservedRunningTime="2026-10-30 06:25:00.566836829 +0000 UTC m=+1914.562793839" Oct 30 06:25:00.763344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:00.747361 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:25:00.939517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:00.939483 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-openshift-ca\") pod \"071940ca-738b-477f-a37b-6f179bc82f8b\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " Oct 30 06:25:00.939858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:00.939814 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/071940ca-738b-477f-a37b-6f179bc82f8b-kube-api-access-2dm57\" (UniqueName: \"kubernetes.io/projected/071940ca-738b-477f-a37b-6f179bc82f8b-kube-api-access-2dm57\") pod \"071940ca-738b-477f-a37b-6f179bc82f8b\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " Oct 30 06:25:00.939966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:00.939947 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-openshift-ca" pod "071940ca-738b-477f-a37b-6f179bc82f8b" (UID: "071940ca-738b-477f-a37b-6f179bc82f8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:00.940052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:00.940037 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-xks-ca\") pod \"071940ca-738b-477f-a37b-6f179bc82f8b\" (UID: \"071940ca-738b-477f-a37b-6f179bc82f8b\") " Oct 30 06:25:00.940652 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:00.940629 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:00.942659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:00.942422 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071940ca-738b-477f-a37b-6f179bc82f8b-kube-api-access-2dm57" pod "071940ca-738b-477f-a37b-6f179bc82f8b" (UID: "071940ca-738b-477f-a37b-6f179bc82f8b"). InnerVolumeSpecName "kube-api-access-2dm57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:00.943441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:00.943097 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-xks-ca" pod "071940ca-738b-477f-a37b-6f179bc82f8b" (UID: "071940ca-738b-477f-a37b-6f179bc82f8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:01.042164 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.042109 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dm57\" (UniqueName: \"kubernetes.io/projected/071940ca-738b-477f-a37b-6f179bc82f8b-kube-api-access-2dm57\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:01.042164 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.042170 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071940ca-738b-477f-a37b-6f179bc82f8b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:01.428451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.428421 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:25:01.428993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.428971 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:25:01.431809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.431785 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:25:01.515362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.515332 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:25:01.520687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.520661 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:25:01.532415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.532387 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" Oct 30 06:25:01.532877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.532382 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz" event={"ID":"071940ca-738b-477f-a37b-6f179bc82f8b","Type":"ContainerDied","Data":"36bb794ac026b2c4d5f64b371e82070b90badec0ed4f2252e16b27f6929b705f"} Oct 30 06:25:01.532877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.532513 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36bb794ac026b2c4d5f64b371e82070b90badec0ed4f2252e16b27f6929b705f" Oct 30 06:25:01.532877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.532870 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:25:01.533992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:01.533974 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:25:02.300233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.300204 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:25:02.381329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.375041 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz"] Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.401352 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-credential-socket\") pod \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.401416 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-kube-api-access-g9dhc\" (UniqueName: \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-kube-api-access-g9dhc\") pod \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.401539 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-certs\") pod \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.401616 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-data\") pod \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.401634 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-credential-socket" pod "10785405-d520-4f7f-9ddc-af67c7c5a1bf" (UID: "10785405-d520-4f7f-9ddc-af67c7c5a1bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.401678 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-podinfo\") pod \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.401820 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-token\" (UniqueName: \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-token\") pod \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.401913 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-socket\") pod \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.401967 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istiod-ca-cert\") pod \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.402020 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-envoy\") pod \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\" (UID: \"10785405-d520-4f7f-9ddc-af67c7c5a1bf\") " Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.402785 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.403600 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-certs" pod "10785405-d520-4f7f-9ddc-af67c7c5a1bf" (UID: "10785405-d520-4f7f-9ddc-af67c7c5a1bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.403607 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-data" pod "10785405-d520-4f7f-9ddc-af67c7c5a1bf" (UID: "10785405-d520-4f7f-9ddc-af67c7c5a1bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.403792 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-socket" pod "10785405-d520-4f7f-9ddc-af67c7c5a1bf" (UID: "10785405-d520-4f7f-9ddc-af67c7c5a1bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:02.404874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.404072 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istiod-ca-cert" pod "10785405-d520-4f7f-9ddc-af67c7c5a1bf" (UID: "10785405-d520-4f7f-9ddc-af67c7c5a1bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:02.418411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.405360 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-kube-api-access-g9dhc" pod "10785405-d520-4f7f-9ddc-af67c7c5a1bf" (UID: "10785405-d520-4f7f-9ddc-af67c7c5a1bf"). InnerVolumeSpecName "kube-api-access-g9dhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:02.418411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.416809 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp"] Oct 30 06:25:02.418870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.418843 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-envoy" pod "10785405-d520-4f7f-9ddc-af67c7c5a1bf" (UID: "10785405-d520-4f7f-9ddc-af67c7c5a1bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:02.420680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.420654 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-podinfo" pod "10785405-d520-4f7f-9ddc-af67c7c5a1bf" (UID: "10785405-d520-4f7f-9ddc-af67c7c5a1bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:25:02.421587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.421547 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-token" pod "10785405-d520-4f7f-9ddc-af67c7c5a1bf" (UID: "10785405-d520-4f7f-9ddc-af67c7c5a1bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:02.505380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.503890 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:02.505380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.503935 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:02.505380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.503954 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:02.505380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.503969 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9dhc\" (UniqueName: \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-kube-api-access-g9dhc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:02.505380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.503985 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:02.505380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.504000 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:02.505380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.504014 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:02.505380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.504028 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10785405-d520-4f7f-9ddc-af67c7c5a1bf-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:02.554478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.554442 5001 generic.go:356] "Generic (PLEG): container finished" podID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerID="03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d" exitCode=0 Oct 30 06:25:02.554968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.554514 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" Oct 30 06:25:02.554968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.554550 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" event={"ID":"10785405-d520-4f7f-9ddc-af67c7c5a1bf","Type":"ContainerDied","Data":"03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d"} Oct 30 06:25:02.554968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.554616 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5" event={"ID":"10785405-d520-4f7f-9ddc-af67c7c5a1bf","Type":"ContainerDied","Data":"96d4c432540d8d26ab53465f8b7eebefe7465ce957bec2ad8d64b58f5e2adbe9"} Oct 30 06:25:02.554968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.554652 5001 scope.go:122] "RemoveContainer" containerID="03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d" Oct 30 06:25:02.569047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.569020 5001 scope.go:122] "RemoveContainer" containerID="7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e" Oct 30 06:25:02.585614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.585574 5001 scope.go:122] "RemoveContainer" containerID="03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d" Oct 30 06:25:02.586055 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:25:02.586032 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d\": container with ID starting with 03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d not found: ID does not exist" containerID="03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d" Oct 30 06:25:02.586163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.586070 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d"} err="failed to get container status \"03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d\": rpc error: code = NotFound desc = could not find container \"03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d\": container with ID starting with 03e0fdee7fa9e251e81d7e89b9443eaad101e56b0b29b721a1a21b571fdea18d not found: ID does not exist" Oct 30 06:25:02.586163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.586100 5001 scope.go:122] "RemoveContainer" containerID="7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e" Oct 30 06:25:02.586503 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:25:02.586481 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e\": container with ID starting with 7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e not found: ID does not exist" containerID="7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e" Oct 30 06:25:02.586568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.586511 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e"} err="failed to get container status \"7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e\": rpc error: code = NotFound desc = could not find container \"7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e\": container with ID starting with 7983635c8d15bc3cb7dce0cb76d50b1d6b3b8b21e03999e374cd2fb77c55eb7e not found: ID does not exist" Oct 30 06:25:02.682897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.682867 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5"] Oct 30 06:25:02.686765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:02.686734 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f2cf8a-openshift-default-7f46c5657c-tjkc5"] Oct 30 06:25:04.118814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:04.118780 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" path="/var/lib/kubelet/pods/10785405-d520-4f7f-9ddc-af67c7c5a1bf/volumes" Oct 30 06:25:05.048933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.048894 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2"] Oct 30 06:25:05.049615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.049585 5001 cpu_manager.go:414] "containerMap: removing container" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" Oct 30 06:25:05.049615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.049611 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" Oct 30 06:25:05.049803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.049630 5001 cpu_manager.go:414] "containerMap: removing container" podUID="071940ca-738b-477f-a37b-6f179bc82f8b" containerName="revoke-keys" Oct 30 06:25:05.049803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.049637 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071940ca-738b-477f-a37b-6f179bc82f8b" containerName="revoke-keys" Oct 30 06:25:05.049803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.049675 5001 cpu_manager.go:414] "containerMap: removing container" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" Oct 30 06:25:05.049803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.049682 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" Oct 30 06:25:05.050016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.049840 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" Oct 30 06:25:05.050016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.049865 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="10785405-d520-4f7f-9ddc-af67c7c5a1bf" containerName="istio-proxy" Oct 30 06:25:05.050016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.049875 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="071940ca-738b-477f-a37b-6f179bc82f8b" containerName="revoke-keys" Oct 30 06:25:05.055073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.055050 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:25:05.076901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.076863 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f5e55f\"" Oct 30 06:25:05.078705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.078682 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2"] Oct 30 06:25:05.085098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.085073 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f5e55f-dockercfg-stbzt\"" Oct 30 06:25:05.197330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.196853 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k"] Oct 30 06:25:05.333646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.314849 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:25:05.362409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.362323 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k"] Oct 30 06:25:05.364854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.364819 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d94784-4586-492e-9f9c-fcea45efb051-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2\" (UID: \"91d94784-4586-492e-9f9c-fcea45efb051\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:25:05.365606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.365579 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rw6t\" (UniqueName: \"kubernetes.io/projected/d869917a-e4ec-4340-a5d9-cac52710f4ec-kube-api-access-7rw6t\") pod \"payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k\" (UID: \"d869917a-e4ec-4340-a5d9-cac52710f4ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:25:05.365728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.365713 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6r7r\" (UniqueName: \"kubernetes.io/projected/91d94784-4586-492e-9f9c-fcea45efb051-kube-api-access-p6r7r\") pod \"payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2\" (UID: \"91d94784-4586-492e-9f9c-fcea45efb051\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:25:05.365896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.365875 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d869917a-e4ec-4340-a5d9-cac52710f4ec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k\" (UID: \"d869917a-e4ec-4340-a5d9-cac52710f4ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:25:05.467667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.467631 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rw6t\" (UniqueName: \"kubernetes.io/projected/d869917a-e4ec-4340-a5d9-cac52710f4ec-kube-api-access-7rw6t\") pod \"payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k\" (UID: \"d869917a-e4ec-4340-a5d9-cac52710f4ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:25:05.467883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.467676 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6r7r\" (UniqueName: \"kubernetes.io/projected/91d94784-4586-492e-9f9c-fcea45efb051-kube-api-access-p6r7r\") pod \"payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2\" (UID: \"91d94784-4586-492e-9f9c-fcea45efb051\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:25:05.467883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.467719 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d869917a-e4ec-4340-a5d9-cac52710f4ec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k\" (UID: \"d869917a-e4ec-4340-a5d9-cac52710f4ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:25:05.467883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.467861 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d94784-4586-492e-9f9c-fcea45efb051-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2\" (UID: \"91d94784-4586-492e-9f9c-fcea45efb051\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:25:05.468578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.468553 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d869917a-e4ec-4340-a5d9-cac52710f4ec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k\" (UID: \"d869917a-e4ec-4340-a5d9-cac52710f4ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:25:05.468676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.468641 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d94784-4586-492e-9f9c-fcea45efb051-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2\" (UID: \"91d94784-4586-492e-9f9c-fcea45efb051\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:25:05.491210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.491182 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rw6t\" (UniqueName: \"kubernetes.io/projected/d869917a-e4ec-4340-a5d9-cac52710f4ec-kube-api-access-7rw6t\") pod \"payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k\" (UID: \"d869917a-e4ec-4340-a5d9-cac52710f4ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:25:05.491412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.491249 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6r7r\" (UniqueName: \"kubernetes.io/projected/91d94784-4586-492e-9f9c-fcea45efb051-kube-api-access-p6r7r\") pod \"payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2\" (UID: \"91d94784-4586-492e-9f9c-fcea45efb051\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:25:05.671671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.671634 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:25:05.673778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:05.673758 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:25:06.065120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:06.065093 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k"] Oct 30 06:25:06.066257 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:25:06.066228 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd869917a_e4ec_4340_a5d9_cac52710f4ec.slice/crio-adbc2dc8f3d8907c6dff0d3d6aded5a53387a55cdc46b82e403af79a1b67fe05 WatchSource:0}: Error finding container adbc2dc8f3d8907c6dff0d3d6aded5a53387a55cdc46b82e403af79a1b67fe05: Status 404 returned error can't find the container with id adbc2dc8f3d8907c6dff0d3d6aded5a53387a55cdc46b82e403af79a1b67fe05 Oct 30 06:25:06.082022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:06.081994 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2"] Oct 30 06:25:06.082397 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:25:06.082375 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d94784_4586_492e_9f9c_fcea45efb051.slice/crio-a19006327ba763278eac6294b5dd010e1065432b649fdfbf7f87643435f40e46 WatchSource:0}: Error finding container a19006327ba763278eac6294b5dd010e1065432b649fdfbf7f87643435f40e46: Status 404 returned error can't find the container with id a19006327ba763278eac6294b5dd010e1065432b649fdfbf7f87643435f40e46 Oct 30 06:25:06.592189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:06.592149 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" event={"ID":"91d94784-4586-492e-9f9c-fcea45efb051","Type":"ContainerStarted","Data":"f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82"} Oct 30 06:25:06.592739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:06.592215 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" event={"ID":"91d94784-4586-492e-9f9c-fcea45efb051","Type":"ContainerStarted","Data":"a19006327ba763278eac6294b5dd010e1065432b649fdfbf7f87643435f40e46"} Oct 30 06:25:06.592739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:06.592245 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:25:06.594376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:06.594349 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" event={"ID":"d869917a-e4ec-4340-a5d9-cac52710f4ec","Type":"ContainerStarted","Data":"bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7"} Oct 30 06:25:06.594499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:06.594397 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" event={"ID":"d869917a-e4ec-4340-a5d9-cac52710f4ec","Type":"ContainerStarted","Data":"adbc2dc8f3d8907c6dff0d3d6aded5a53387a55cdc46b82e403af79a1b67fe05"} Oct 30 06:25:06.594499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:06.594425 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:25:06.633734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:06.633677 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" podStartSLOduration=2.63366326 podStartE2EDuration="2.63366326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:06.630466441 +0000 UTC m=+1920.626423450" watchObservedRunningTime="2026-10-30 06:25:06.63366326 +0000 UTC m=+1920.629620269" Oct 30 06:25:06.670589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:06.670532 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" podStartSLOduration=1.670517926 podStartE2EDuration="1.670517926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:06.669830528 +0000 UTC m=+1920.665787540" watchObservedRunningTime="2026-10-30 06:25:06.670517926 +0000 UTC m=+1920.666474935" Oct 30 06:25:07.376245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:07.376198 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" podUID="3bf6d2cb-7ed8-4375-b0da-73a49945b0fb" containerName="payload-processing" containerID="cri-o://29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba" gracePeriod=25 Oct 30 06:25:07.417915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:07.417877 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" podUID="dc4c812d-c16f-4027-8413-f364014c1f1b" containerName="payload-pre-processing" containerID="cri-o://4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab" gracePeriod=25 Oct 30 06:25:08.269205 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:25:08.269167 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:25:08.593253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:08.593174 5001 scope.go:122] "RemoveContainer" containerID="0b2436c470a9ef9d3a545c1bfc2e9e062ff5b578da4cc710cc199450e656afb9" Oct 30 06:25:09.466057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.466015 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" podUID="dc4c812d-c16f-4027-8413-f364014c1f1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:25:09.466609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.466029 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" podUID="3bf6d2cb-7ed8-4375-b0da-73a49945b0fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:25:09.895779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.895742 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns"] Oct 30 06:25:09.901928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.901904 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:09.930075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.930043 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:25:09.930075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.930067 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:25:09.930402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.930067 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:25:09.930402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.930211 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:25:09.935010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.934976 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns"] Oct 30 06:25:09.959414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.959364 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75713d4c-x5fns\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:09.959612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.959514 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75713d4c-x5fns\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:09.959657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:09.959640 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4fvd\" (UniqueName: \"kubernetes.io/projected/843533bb-ca51-450c-b95d-557df4ee6423-kube-api-access-c4fvd\") pod \"maas-api-revoke-sub-75713d4c-x5fns\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:10.060927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.060895 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4fvd\" (UniqueName: \"kubernetes.io/projected/843533bb-ca51-450c-b95d-557df4ee6423-kube-api-access-c4fvd\") pod \"maas-api-revoke-sub-75713d4c-x5fns\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:10.061145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.061040 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75713d4c-x5fns\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:10.061145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.061130 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75713d4c-x5fns\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:10.062003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.061977 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75713d4c-x5fns\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:10.063730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.063702 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75713d4c-x5fns\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:10.083965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.083928 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4fvd\" (UniqueName: \"kubernetes.io/projected/843533bb-ca51-450c-b95d-557df4ee6423-kube-api-access-c4fvd\") pod \"maas-api-revoke-sub-75713d4c-x5fns\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:10.217284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.217178 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:10.536490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.536466 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns"] Oct 30 06:25:10.538472 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:25:10.538443 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843533bb_ca51_450c_b95d_557df4ee6423.slice/crio-f309cd2262e3f85ecd9aba405087983c4ec90769386c908228254807b118d1bd WatchSource:0}: Error finding container f309cd2262e3f85ecd9aba405087983c4ec90769386c908228254807b118d1bd: Status 404 returned error can't find the container with id f309cd2262e3f85ecd9aba405087983c4ec90769386c908228254807b118d1bd Oct 30 06:25:10.641010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.640972 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" event={"ID":"843533bb-ca51-450c-b95d-557df4ee6423","Type":"ContainerStarted","Data":"f309cd2262e3f85ecd9aba405087983c4ec90769386c908228254807b118d1bd"} Oct 30 06:25:10.913558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.913524 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs"] Oct 30 06:25:10.917792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.917776 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:10.944141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.944111 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs"] Oct 30 06:25:10.974294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.974257 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784206f4-hqmhs\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:10.974521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.974502 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784206f4-hqmhs\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:10.974778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:10.974685 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wkv\" (UniqueName: \"kubernetes.io/projected/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-kube-api-access-99wkv\") pod \"maas-api-revoke-sub-784206f4-hqmhs\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:11.075667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:11.075627 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784206f4-hqmhs\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:11.075837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:11.075794 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wkv\" (UniqueName: \"kubernetes.io/projected/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-kube-api-access-99wkv\") pod \"maas-api-revoke-sub-784206f4-hqmhs\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:11.075940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:11.075923 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784206f4-hqmhs\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:11.076575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:11.076548 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784206f4-hqmhs\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:11.078234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:11.078214 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784206f4-hqmhs\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:11.099725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:11.099690 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wkv\" (UniqueName: \"kubernetes.io/projected/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-kube-api-access-99wkv\") pod \"maas-api-revoke-sub-784206f4-hqmhs\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:11.232476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:11.232399 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:11.560578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:11.560515 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs"] Oct 30 06:25:11.564014 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:25:11.563986 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5624a5_bb95_4c59_b1d4_0cf4d60443eb.slice/crio-91f63da826582ac09e81359bf1a7d0764c9952955cabed98503cc034b5b951c0 WatchSource:0}: Error finding container 91f63da826582ac09e81359bf1a7d0764c9952955cabed98503cc034b5b951c0: Status 404 returned error can't find the container with id 91f63da826582ac09e81359bf1a7d0764c9952955cabed98503cc034b5b951c0 Oct 30 06:25:11.705377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:11.705286 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" event={"ID":"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb","Type":"ContainerStarted","Data":"91f63da826582ac09e81359bf1a7d0764c9952955cabed98503cc034b5b951c0"} Oct 30 06:25:11.707251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:11.707222 5001 generic.go:356] "Generic (PLEG): container finished" podID="843533bb-ca51-450c-b95d-557df4ee6423" containerID="743db714948c8407ab0b1dd12246f84d84e2227ae95571d5aca7ca0a9283ed8d" exitCode=0 Oct 30 06:25:11.707502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:11.707321 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" event={"ID":"843533bb-ca51-450c-b95d-557df4ee6423","Type":"ContainerDied","Data":"743db714948c8407ab0b1dd12246f84d84e2227ae95571d5aca7ca0a9283ed8d"} Oct 30 06:25:12.337411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:12.337331 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:25:12.717711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:12.717675 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" event={"ID":"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb","Type":"ContainerStarted","Data":"7d4c89a6d8e60b8494ea5a3e1d84f85df46c2e28ab0bfdb6a0c26300b8f9539d"} Oct 30 06:25:12.763509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:12.763443 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" podStartSLOduration=2.7634248120000002 podStartE2EDuration="2.763424812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:12.759081175 +0000 UTC m=+1926.755038184" watchObservedRunningTime="2026-10-30 06:25:12.763424812 +0000 UTC m=+1926.759381823" Oct 30 06:25:12.979780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:12.979712 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:13.102421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.102388 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843533bb-ca51-450c-b95d-557df4ee6423-kube-api-access-c4fvd\" (UniqueName: \"kubernetes.io/projected/843533bb-ca51-450c-b95d-557df4ee6423-kube-api-access-c4fvd\") pod \"843533bb-ca51-450c-b95d-557df4ee6423\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " Oct 30 06:25:13.102624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.102444 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-openshift-ca\") pod \"843533bb-ca51-450c-b95d-557df4ee6423\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " Oct 30 06:25:13.102816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.102781 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-xks-ca\") pod \"843533bb-ca51-450c-b95d-557df4ee6423\" (UID: \"843533bb-ca51-450c-b95d-557df4ee6423\") " Oct 30 06:25:13.102953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.102841 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-openshift-ca" pod "843533bb-ca51-450c-b95d-557df4ee6423" (UID: "843533bb-ca51-450c-b95d-557df4ee6423"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:13.103266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.103247 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:13.104816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.104793 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843533bb-ca51-450c-b95d-557df4ee6423-kube-api-access-c4fvd" pod "843533bb-ca51-450c-b95d-557df4ee6423" (UID: "843533bb-ca51-450c-b95d-557df4ee6423"). InnerVolumeSpecName "kube-api-access-c4fvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:13.104926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.104845 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-xks-ca" pod "843533bb-ca51-450c-b95d-557df4ee6423" (UID: "843533bb-ca51-450c-b95d-557df4ee6423"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:13.246380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.238431 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843533bb-ca51-450c-b95d-557df4ee6423-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:13.246380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.238469 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4fvd\" (UniqueName: \"kubernetes.io/projected/843533bb-ca51-450c-b95d-557df4ee6423-kube-api-access-c4fvd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:13.774850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.774810 5001 generic.go:356] "Generic (PLEG): container finished" podID="ac5624a5-bb95-4c59-b1d4-0cf4d60443eb" containerID="7d4c89a6d8e60b8494ea5a3e1d84f85df46c2e28ab0bfdb6a0c26300b8f9539d" exitCode=0 Oct 30 06:25:13.775546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.775519 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" event={"ID":"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb","Type":"ContainerDied","Data":"7d4c89a6d8e60b8494ea5a3e1d84f85df46c2e28ab0bfdb6a0c26300b8f9539d"} Oct 30 06:25:13.796499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.796462 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" event={"ID":"843533bb-ca51-450c-b95d-557df4ee6423","Type":"ContainerDied","Data":"f309cd2262e3f85ecd9aba405087983c4ec90769386c908228254807b118d1bd"} Oct 30 06:25:13.796813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.796796 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f309cd2262e3f85ecd9aba405087983c4ec90769386c908228254807b118d1bd" Oct 30 06:25:13.796941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:13.796485 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns" Oct 30 06:25:15.159593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.159567 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:15.238842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.238754 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-xks-ca\") pod \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " Oct 30 06:25:15.239077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.239004 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-kube-api-access-99wkv\" (UniqueName: \"kubernetes.io/projected/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-kube-api-access-99wkv\") pod \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " Oct 30 06:25:15.239231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.239098 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-openshift-ca\") pod \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\" (UID: \"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb\") " Oct 30 06:25:15.239494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.239465 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-openshift-ca" pod "ac5624a5-bb95-4c59-b1d4-0cf4d60443eb" (UID: "ac5624a5-bb95-4c59-b1d4-0cf4d60443eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:15.239665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.239647 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:15.241684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.241657 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-xks-ca" pod "ac5624a5-bb95-4c59-b1d4-0cf4d60443eb" (UID: "ac5624a5-bb95-4c59-b1d4-0cf4d60443eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:15.241810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.241791 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-kube-api-access-99wkv" pod "ac5624a5-bb95-4c59-b1d4-0cf4d60443eb" (UID: "ac5624a5-bb95-4c59-b1d4-0cf4d60443eb"). InnerVolumeSpecName "kube-api-access-99wkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:15.314631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.314595 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj"] Oct 30 06:25:15.315275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.315256 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ac5624a5-bb95-4c59-b1d4-0cf4d60443eb" containerName="revoke-keys" Oct 30 06:25:15.315410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.315277 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5624a5-bb95-4c59-b1d4-0cf4d60443eb" containerName="revoke-keys" Oct 30 06:25:15.315410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.315354 5001 cpu_manager.go:414] "containerMap: removing container" podUID="843533bb-ca51-450c-b95d-557df4ee6423" containerName="revoke-keys" Oct 30 06:25:15.315410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.315364 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843533bb-ca51-450c-b95d-557df4ee6423" containerName="revoke-keys" Oct 30 06:25:15.315575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.315539 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="843533bb-ca51-450c-b95d-557df4ee6423" containerName="revoke-keys" Oct 30 06:25:15.315575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.315560 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5624a5-bb95-4c59-b1d4-0cf4d60443eb" containerName="revoke-keys" Oct 30 06:25:15.319774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.319752 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:15.340701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.340665 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t7wt\" (UniqueName: \"kubernetes.io/projected/f5e5df40-aa16-4408-820e-11bcafd55ec0-kube-api-access-7t7wt\") pod \"maas-api-revoke-sub-e0f3c7a3-zdrzj\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:15.340886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.340758 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f3c7a3-zdrzj\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:15.340969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.340889 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f3c7a3-zdrzj\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:15.340969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.340952 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99wkv\" (UniqueName: \"kubernetes.io/projected/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-kube-api-access-99wkv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:15.341093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.340968 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:15.342341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.342291 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj"] Oct 30 06:25:15.441932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.441892 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f3c7a3-zdrzj\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:15.442124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.442033 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t7wt\" (UniqueName: \"kubernetes.io/projected/f5e5df40-aa16-4408-820e-11bcafd55ec0-kube-api-access-7t7wt\") pod \"maas-api-revoke-sub-e0f3c7a3-zdrzj\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:15.442124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.442103 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f3c7a3-zdrzj\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:15.442844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.442785 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f3c7a3-zdrzj\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:15.444644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.444618 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f3c7a3-zdrzj\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:15.466112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.466083 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t7wt\" (UniqueName: \"kubernetes.io/projected/f5e5df40-aa16-4408-820e-11bcafd55ec0-kube-api-access-7t7wt\") pod \"maas-api-revoke-sub-e0f3c7a3-zdrzj\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:15.637641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.637610 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:15.967351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.961513 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj"] Oct 30 06:25:15.972360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.969410 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" Oct 30 06:25:15.972360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.969428 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs" event={"ID":"ac5624a5-bb95-4c59-b1d4-0cf4d60443eb","Type":"ContainerDied","Data":"91f63da826582ac09e81359bf1a7d0764c9952955cabed98503cc034b5b951c0"} Oct 30 06:25:15.972360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:15.969492 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91f63da826582ac09e81359bf1a7d0764c9952955cabed98503cc034b5b951c0" Oct 30 06:25:16.981114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:16.981079 5001 generic.go:356] "Generic (PLEG): container finished" podID="f5e5df40-aa16-4408-820e-11bcafd55ec0" containerID="731a899c21b1d73174edc68af0b62982b3812e66d35e55ea621dcbb3ddb5be95" exitCode=0 Oct 30 06:25:16.981542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:16.981173 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" event={"ID":"f5e5df40-aa16-4408-820e-11bcafd55ec0","Type":"ContainerDied","Data":"731a899c21b1d73174edc68af0b62982b3812e66d35e55ea621dcbb3ddb5be95"} Oct 30 06:25:16.981542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:16.981241 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" event={"ID":"f5e5df40-aa16-4408-820e-11bcafd55ec0","Type":"ContainerStarted","Data":"32f28ea2bd066cae05700078c98a97aa1b577512efafb72de4f043a4435c6efe"} Oct 30 06:25:17.607282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:17.607245 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:25:17.607878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:17.607861 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:25:18.206787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:18.206758 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:18.296384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:18.296279 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-openshift-ca\") pod \"f5e5df40-aa16-4408-820e-11bcafd55ec0\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " Oct 30 06:25:18.296661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:18.296636 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-xks-ca\") pod \"f5e5df40-aa16-4408-820e-11bcafd55ec0\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " Oct 30 06:25:18.296848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:18.296834 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e5df40-aa16-4408-820e-11bcafd55ec0-kube-api-access-7t7wt\" (UniqueName: \"kubernetes.io/projected/f5e5df40-aa16-4408-820e-11bcafd55ec0-kube-api-access-7t7wt\") pod \"f5e5df40-aa16-4408-820e-11bcafd55ec0\" (UID: \"f5e5df40-aa16-4408-820e-11bcafd55ec0\") " Oct 30 06:25:18.298531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:18.298505 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-openshift-ca" pod "f5e5df40-aa16-4408-820e-11bcafd55ec0" (UID: "f5e5df40-aa16-4408-820e-11bcafd55ec0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:18.320835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:18.320518 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-xks-ca" pod "f5e5df40-aa16-4408-820e-11bcafd55ec0" (UID: "f5e5df40-aa16-4408-820e-11bcafd55ec0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:18.381355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:18.380978 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e5df40-aa16-4408-820e-11bcafd55ec0-kube-api-access-7t7wt" pod "f5e5df40-aa16-4408-820e-11bcafd55ec0" (UID: "f5e5df40-aa16-4408-820e-11bcafd55ec0"). InnerVolumeSpecName "kube-api-access-7t7wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:18.399186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:18.399147 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:18.399186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:18.399176 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e5df40-aa16-4408-820e-11bcafd55ec0-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:18.399186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:18.399186 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t7wt\" (UniqueName: \"kubernetes.io/projected/f5e5df40-aa16-4408-820e-11bcafd55ec0-kube-api-access-7t7wt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:19.003233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:19.003195 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" Oct 30 06:25:19.003233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:19.003226 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj" event={"ID":"f5e5df40-aa16-4408-820e-11bcafd55ec0","Type":"ContainerDied","Data":"32f28ea2bd066cae05700078c98a97aa1b577512efafb72de4f043a4435c6efe"} Oct 30 06:25:19.003527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:19.003279 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32f28ea2bd066cae05700078c98a97aa1b577512efafb72de4f043a4435c6efe" Oct 30 06:25:19.466296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:19.466255 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" podUID="dc4c812d-c16f-4027-8413-f364014c1f1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:25:19.466803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:19.466259 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" podUID="3bf6d2cb-7ed8-4375-b0da-73a49945b0fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:25:27.402652 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.402625 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7"] Oct 30 06:25:27.403214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.403200 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f5e5df40-aa16-4408-820e-11bcafd55ec0" containerName="revoke-keys" Oct 30 06:25:27.403279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.403216 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e5df40-aa16-4408-820e-11bcafd55ec0" containerName="revoke-keys" Oct 30 06:25:27.403375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.403365 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e5df40-aa16-4408-820e-11bcafd55ec0" containerName="revoke-keys" Oct 30 06:25:27.406875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.406856 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:27.423864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.423829 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:25:27.424587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.424197 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:25:27.429518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.429488 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7"] Oct 30 06:25:27.438933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.438908 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:25:27.439139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.438933 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:25:27.547239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.547209 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwrz4\" (UniqueName: \"kubernetes.io/projected/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-kube-api-access-jwrz4\") pod \"maas-api-revoke-sub-66f3ebe5-g58k7\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:27.547461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.547380 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66f3ebe5-g58k7\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:27.547461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.547455 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66f3ebe5-g58k7\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:27.650590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.649511 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwrz4\" (UniqueName: \"kubernetes.io/projected/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-kube-api-access-jwrz4\") pod \"maas-api-revoke-sub-66f3ebe5-g58k7\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:27.650590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.649635 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66f3ebe5-g58k7\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:27.650590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.649704 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66f3ebe5-g58k7\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:27.651618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.651568 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66f3ebe5-g58k7\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:27.672731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.672667 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66f3ebe5-g58k7\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:27.682819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.682786 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwrz4\" (UniqueName: \"kubernetes.io/projected/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-kube-api-access-jwrz4\") pod \"maas-api-revoke-sub-66f3ebe5-g58k7\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:27.722954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.722928 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:25:27.733752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.733728 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:25:27.844532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.844496 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:27.851891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.851865 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-kube-api-access-hpvct\" (UniqueName: \"kubernetes.io/projected/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-kube-api-access-hpvct\") pod \"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb\" (UID: \"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb\") " Oct 30 06:25:27.852009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.851992 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-plugins-config-volume\") pod \"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb\" (UID: \"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb\") " Oct 30 06:25:27.852429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.852403 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-plugins-config-volume" pod "3bf6d2cb-7ed8-4375-b0da-73a49945b0fb" (UID: "3bf6d2cb-7ed8-4375-b0da-73a49945b0fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:27.852642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.852596 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4c812d-c16f-4027-8413-f364014c1f1b-kube-api-access-zm2c5\" (UniqueName: \"kubernetes.io/projected/dc4c812d-c16f-4027-8413-f364014c1f1b-kube-api-access-zm2c5\") pod \"dc4c812d-c16f-4027-8413-f364014c1f1b\" (UID: \"dc4c812d-c16f-4027-8413-f364014c1f1b\") " Oct 30 06:25:27.852642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.852654 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4c812d-c16f-4027-8413-f364014c1f1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4c812d-c16f-4027-8413-f364014c1f1b-plugins-config-volume\") pod \"dc4c812d-c16f-4027-8413-f364014c1f1b\" (UID: \"dc4c812d-c16f-4027-8413-f364014c1f1b\") " Oct 30 06:25:27.853352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.853326 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:27.853493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.853466 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4c812d-c16f-4027-8413-f364014c1f1b-plugins-config-volume" pod "dc4c812d-c16f-4027-8413-f364014c1f1b" (UID: "dc4c812d-c16f-4027-8413-f364014c1f1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:27.854505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.854480 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-kube-api-access-hpvct" pod "3bf6d2cb-7ed8-4375-b0da-73a49945b0fb" (UID: "3bf6d2cb-7ed8-4375-b0da-73a49945b0fb"). InnerVolumeSpecName "kube-api-access-hpvct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:27.854918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.854895 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4c812d-c16f-4027-8413-f364014c1f1b-kube-api-access-zm2c5" pod "dc4c812d-c16f-4027-8413-f364014c1f1b" (UID: "dc4c812d-c16f-4027-8413-f364014c1f1b"). InnerVolumeSpecName "kube-api-access-zm2c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:28.003822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.996597 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm2c5\" (UniqueName: \"kubernetes.io/projected/dc4c812d-c16f-4027-8413-f364014c1f1b-kube-api-access-zm2c5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:28.003822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.996678 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4c812d-c16f-4027-8413-f364014c1f1b-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:28.003822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:27.996714 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpvct\" (UniqueName: \"kubernetes.io/projected/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb-kube-api-access-hpvct\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:28.097569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.097531 5001 generic.go:356] "Generic (PLEG): container finished" podID="3bf6d2cb-7ed8-4375-b0da-73a49945b0fb" containerID="29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba" exitCode=0 Oct 30 06:25:28.097794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.097589 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" Oct 30 06:25:28.097794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.097611 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" event={"ID":"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb","Type":"ContainerDied","Data":"29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba"} Oct 30 06:25:28.097794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.097671 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz" event={"ID":"3bf6d2cb-7ed8-4375-b0da-73a49945b0fb","Type":"ContainerDied","Data":"dd77f57fc67ef51d8320524f886215e441ddeba4a0cfc0b362098866939f31f5"} Oct 30 06:25:28.097794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.097702 5001 scope.go:122] "RemoveContainer" containerID="29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba" Oct 30 06:25:28.099628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.099604 5001 generic.go:356] "Generic (PLEG): container finished" podID="dc4c812d-c16f-4027-8413-f364014c1f1b" containerID="4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab" exitCode=0 Oct 30 06:25:28.099741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.099646 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" event={"ID":"dc4c812d-c16f-4027-8413-f364014c1f1b","Type":"ContainerDied","Data":"4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab"} Oct 30 06:25:28.099741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.099675 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" event={"ID":"dc4c812d-c16f-4027-8413-f364014c1f1b","Type":"ContainerDied","Data":"4890dcab0438dbca20240c50c5e306209523f32bb0a68059b590fc357654bebe"} Oct 30 06:25:28.099741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.099725 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp" Oct 30 06:25:28.112962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.112935 5001 scope.go:122] "RemoveContainer" containerID="29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba" Oct 30 06:25:28.113275 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:25:28.113239 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba\": container with ID starting with 29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba not found: ID does not exist" containerID="29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba" Oct 30 06:25:28.113395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.113269 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba"} err="failed to get container status \"29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba\": rpc error: code = NotFound desc = could not find container \"29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba\": container with ID starting with 29ed1767750f38b02464a9ccaaa975823ac1cbc50fc1a46c8fef2fb5556df4ba not found: ID does not exist" Oct 30 06:25:28.113395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.113293 5001 scope.go:122] "RemoveContainer" containerID="4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab" Oct 30 06:25:28.128667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.128638 5001 scope.go:122] "RemoveContainer" containerID="4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab" Oct 30 06:25:28.134642 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:25:28.134612 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab\": container with ID starting with 4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab not found: ID does not exist" containerID="4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab" Oct 30 06:25:28.134788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.134652 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab"} err="failed to get container status \"4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab\": rpc error: code = NotFound desc = could not find container \"4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab\": container with ID starting with 4ea2f387ccd221970f4f11adffe689cdcf95d3f3235fe9208a3618708e9a53ab not found: ID does not exist" Oct 30 06:25:28.172678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.172648 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7"] Oct 30 06:25:28.174130 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:25:28.174083 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2d840f_fe44_4329_97c8_b76dfb2ed04e.slice/crio-419a4c5ccd8d7c5e0d09802c31897b5ae673715931477b527ae2dcac54dacbf5 WatchSource:0}: Error finding container 419a4c5ccd8d7c5e0d09802c31897b5ae673715931477b527ae2dcac54dacbf5: Status 404 returned error can't find the container with id 419a4c5ccd8d7c5e0d09802c31897b5ae673715931477b527ae2dcac54dacbf5 Oct 30 06:25:28.179154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.179131 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz"] Oct 30 06:25:28.184867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.184836 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfa2ae-7dcff65486-zmfxz"] Oct 30 06:25:28.218428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.218402 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp"] Oct 30 06:25:28.223611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.223585 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfa2ae-b68b85d4c-2h2tp"] Oct 30 06:25:28.920934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.920902 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg"] Oct 30 06:25:28.921591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.921535 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dc4c812d-c16f-4027-8413-f364014c1f1b" containerName="payload-pre-processing" Oct 30 06:25:28.921591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.921549 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4c812d-c16f-4027-8413-f364014c1f1b" containerName="payload-pre-processing" Oct 30 06:25:28.921591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.921573 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3bf6d2cb-7ed8-4375-b0da-73a49945b0fb" containerName="payload-processing" Oct 30 06:25:28.921591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.921579 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf6d2cb-7ed8-4375-b0da-73a49945b0fb" containerName="payload-processing" Oct 30 06:25:28.921752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.921681 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf6d2cb-7ed8-4375-b0da-73a49945b0fb" containerName="payload-processing" Oct 30 06:25:28.921752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.921696 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4c812d-c16f-4027-8413-f364014c1f1b" containerName="payload-pre-processing" Oct 30 06:25:28.925315 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.925287 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:28.945277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:28.945247 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg"] Oct 30 06:25:29.008821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.008789 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rwg\" (UniqueName: \"kubernetes.io/projected/3912f258-783d-40cb-985a-c7c18a333e4c-kube-api-access-x6rwg\") pod \"maas-api-revoke-sub-4d3a2255-25zvg\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:29.009012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.008956 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d3a2255-25zvg\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:29.009012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.009008 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d3a2255-25zvg\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:29.109772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.109729 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d3a2255-25zvg\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:29.109966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.109830 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rwg\" (UniqueName: \"kubernetes.io/projected/3912f258-783d-40cb-985a-c7c18a333e4c-kube-api-access-x6rwg\") pod \"maas-api-revoke-sub-4d3a2255-25zvg\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:29.109966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.109855 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" event={"ID":"2d2d840f-fe44-4329-97c8-b76dfb2ed04e","Type":"ContainerStarted","Data":"e2005bf97877b18ea7160051545e04d0dda057c8e14c995899d1d9d27c343838"} Oct 30 06:25:29.109966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.109899 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" event={"ID":"2d2d840f-fe44-4329-97c8-b76dfb2ed04e","Type":"ContainerStarted","Data":"419a4c5ccd8d7c5e0d09802c31897b5ae673715931477b527ae2dcac54dacbf5"} Oct 30 06:25:29.110131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.110013 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d3a2255-25zvg\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:29.110511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.110481 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d3a2255-25zvg\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:29.112906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.112883 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d3a2255-25zvg\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:29.133014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.132988 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rwg\" (UniqueName: \"kubernetes.io/projected/3912f258-783d-40cb-985a-c7c18a333e4c-kube-api-access-x6rwg\") pod \"maas-api-revoke-sub-4d3a2255-25zvg\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:29.153683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.153633 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" podStartSLOduration=2.153620298 podStartE2EDuration="2.153620298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:29.149363025 +0000 UTC m=+1943.145320031" watchObservedRunningTime="2026-10-30 06:25:29.153620298 +0000 UTC m=+1943.149577307" Oct 30 06:25:29.240772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.240744 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:29.559677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:29.559650 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg"] Oct 30 06:25:29.562737 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:25:29.562705 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3912f258_783d_40cb_985a_c7c18a333e4c.slice/crio-00ad929e4956c7eddf81945c883dc8a41171ae0ed0db6fc4bb0e48967d47c93c WatchSource:0}: Error finding container 00ad929e4956c7eddf81945c883dc8a41171ae0ed0db6fc4bb0e48967d47c93c: Status 404 returned error can't find the container with id 00ad929e4956c7eddf81945c883dc8a41171ae0ed0db6fc4bb0e48967d47c93c Oct 30 06:25:30.117587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:30.117550 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf6d2cb-7ed8-4375-b0da-73a49945b0fb" path="/var/lib/kubelet/pods/3bf6d2cb-7ed8-4375-b0da-73a49945b0fb/volumes" Oct 30 06:25:30.118043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:30.118027 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4c812d-c16f-4027-8413-f364014c1f1b" path="/var/lib/kubelet/pods/dc4c812d-c16f-4027-8413-f364014c1f1b/volumes" Oct 30 06:25:30.122706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:30.122684 5001 generic.go:356] "Generic (PLEG): container finished" podID="2d2d840f-fe44-4329-97c8-b76dfb2ed04e" containerID="e2005bf97877b18ea7160051545e04d0dda057c8e14c995899d1d9d27c343838" exitCode=0 Oct 30 06:25:30.122861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:30.122721 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" event={"ID":"2d2d840f-fe44-4329-97c8-b76dfb2ed04e","Type":"ContainerDied","Data":"e2005bf97877b18ea7160051545e04d0dda057c8e14c995899d1d9d27c343838"} Oct 30 06:25:30.124529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:30.124498 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" event={"ID":"3912f258-783d-40cb-985a-c7c18a333e4c","Type":"ContainerStarted","Data":"9ab955072530ec810cae1b271d81b59e37b55aa6fe6ea5976af6ffd7dc24ec49"} Oct 30 06:25:30.124652 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:30.124550 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" event={"ID":"3912f258-783d-40cb-985a-c7c18a333e4c","Type":"ContainerStarted","Data":"00ad929e4956c7eddf81945c883dc8a41171ae0ed0db6fc4bb0e48967d47c93c"} Oct 30 06:25:30.209071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:30.209025 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" podStartSLOduration=2.209011707 podStartE2EDuration="2.209011707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:30.20852747 +0000 UTC m=+1944.204484480" watchObservedRunningTime="2026-10-30 06:25:30.209011707 +0000 UTC m=+1944.204968716" Oct 30 06:25:31.134638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.134600 5001 generic.go:356] "Generic (PLEG): container finished" podID="3912f258-783d-40cb-985a-c7c18a333e4c" containerID="9ab955072530ec810cae1b271d81b59e37b55aa6fe6ea5976af6ffd7dc24ec49" exitCode=0 Oct 30 06:25:31.135091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.134688 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" event={"ID":"3912f258-783d-40cb-985a-c7c18a333e4c","Type":"ContainerDied","Data":"9ab955072530ec810cae1b271d81b59e37b55aa6fe6ea5976af6ffd7dc24ec49"} Oct 30 06:25:31.341840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.341808 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:31.441604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.441506 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-xks-ca\") pod \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " Oct 30 06:25:31.441789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.441641 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-openshift-ca\") pod \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " Oct 30 06:25:31.441789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.441665 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-kube-api-access-jwrz4\" (UniqueName: \"kubernetes.io/projected/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-kube-api-access-jwrz4\") pod \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\" (UID: \"2d2d840f-fe44-4329-97c8-b76dfb2ed04e\") " Oct 30 06:25:31.442008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.441985 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-openshift-ca" pod "2d2d840f-fe44-4329-97c8-b76dfb2ed04e" (UID: "2d2d840f-fe44-4329-97c8-b76dfb2ed04e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:31.442159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.442140 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:31.443970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.443945 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-xks-ca" pod "2d2d840f-fe44-4329-97c8-b76dfb2ed04e" (UID: "2d2d840f-fe44-4329-97c8-b76dfb2ed04e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:31.444215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.444197 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-kube-api-access-jwrz4" pod "2d2d840f-fe44-4329-97c8-b76dfb2ed04e" (UID: "2d2d840f-fe44-4329-97c8-b76dfb2ed04e"). InnerVolumeSpecName "kube-api-access-jwrz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:31.557349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.545172 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwrz4\" (UniqueName: \"kubernetes.io/projected/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-kube-api-access-jwrz4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:31.557349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:31.545210 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2d840f-fe44-4329-97c8-b76dfb2ed04e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.144218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.144193 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" Oct 30 06:25:32.144619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.144235 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7" event={"ID":"2d2d840f-fe44-4329-97c8-b76dfb2ed04e","Type":"ContainerDied","Data":"419a4c5ccd8d7c5e0d09802c31897b5ae673715931477b527ae2dcac54dacbf5"} Oct 30 06:25:32.144619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.144333 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="419a4c5ccd8d7c5e0d09802c31897b5ae673715931477b527ae2dcac54dacbf5" Oct 30 06:25:32.327794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.327769 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:32.354779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.354748 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-xks-ca\") pod \"3912f258-783d-40cb-985a-c7c18a333e4c\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " Oct 30 06:25:32.354944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.354917 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3912f258-783d-40cb-985a-c7c18a333e4c-kube-api-access-x6rwg\" (UniqueName: \"kubernetes.io/projected/3912f258-783d-40cb-985a-c7c18a333e4c-kube-api-access-x6rwg\") pod \"3912f258-783d-40cb-985a-c7c18a333e4c\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " Oct 30 06:25:32.355009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.354983 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-openshift-ca\") pod \"3912f258-783d-40cb-985a-c7c18a333e4c\" (UID: \"3912f258-783d-40cb-985a-c7c18a333e4c\") " Oct 30 06:25:32.355464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.355427 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-openshift-ca" pod "3912f258-783d-40cb-985a-c7c18a333e4c" (UID: "3912f258-783d-40cb-985a-c7c18a333e4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:32.355832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.355810 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.357095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.357067 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-xks-ca" pod "3912f258-783d-40cb-985a-c7c18a333e4c" (UID: "3912f258-783d-40cb-985a-c7c18a333e4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:32.357368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.357344 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3912f258-783d-40cb-985a-c7c18a333e4c-kube-api-access-x6rwg" pod "3912f258-783d-40cb-985a-c7c18a333e4c" (UID: "3912f258-783d-40cb-985a-c7c18a333e4c"). InnerVolumeSpecName "kube-api-access-x6rwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:32.457542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.457460 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3912f258-783d-40cb-985a-c7c18a333e4c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.457542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:32.457498 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6rwg\" (UniqueName: \"kubernetes.io/projected/3912f258-783d-40cb-985a-c7c18a333e4c-kube-api-access-x6rwg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:33.154471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:33.154439 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" Oct 30 06:25:33.154949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:33.154478 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg" event={"ID":"3912f258-783d-40cb-985a-c7c18a333e4c","Type":"ContainerDied","Data":"00ad929e4956c7eddf81945c883dc8a41171ae0ed0db6fc4bb0e48967d47c93c"} Oct 30 06:25:33.154949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:33.154532 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00ad929e4956c7eddf81945c883dc8a41171ae0ed0db6fc4bb0e48967d47c93c" Oct 30 06:25:33.873676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:33.873638 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7"] Oct 30 06:25:33.885456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:33.885424 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfa2ae-9vqw7"] Oct 30 06:25:34.117868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:34.117831 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b" path="/var/lib/kubelet/pods/6aec366f-9c24-4dc7-b1bf-de7ebeefcb8b/volumes" Oct 30 06:25:37.672329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.664567 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766"] Oct 30 06:25:37.672329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.665525 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3912f258-783d-40cb-985a-c7c18a333e4c" containerName="revoke-keys" Oct 30 06:25:37.672329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.665550 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3912f258-783d-40cb-985a-c7c18a333e4c" containerName="revoke-keys" Oct 30 06:25:37.672329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.665575 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2d2d840f-fe44-4329-97c8-b76dfb2ed04e" containerName="revoke-keys" Oct 30 06:25:37.672329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.665582 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2d840f-fe44-4329-97c8-b76dfb2ed04e" containerName="revoke-keys" Oct 30 06:25:37.672329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.665790 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3912f258-783d-40cb-985a-c7c18a333e4c" containerName="revoke-keys" Oct 30 06:25:37.672329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.665808 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2d840f-fe44-4329-97c8-b76dfb2ed04e" containerName="revoke-keys" Oct 30 06:25:37.698553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.698520 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766"] Oct 30 06:25:37.698900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.698883 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.740341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.736643 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:25:37.809342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.800520 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klj49\" (UniqueName: \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-kube-api-access-klj49\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.809342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.800584 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.809342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.800676 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.809342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.800712 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.809342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.800746 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.809342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.800798 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a380d4-e404-41ed-9e49-1af4f8b822d8-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.809342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.800867 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.809342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.800891 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.809342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.800920 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.947401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.905115 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.962490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.962433 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.971442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.971403 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.991427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.967499 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:37.991703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:37.905923 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.008367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.003606 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a380d4-e404-41ed-9e49-1af4f8b822d8-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.008367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.003780 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.008367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.003826 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.008367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.003863 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.008367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.004026 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klj49\" (UniqueName: \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-kube-api-access-klj49\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.008367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.004096 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.035335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.009609 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a380d4-e404-41ed-9e49-1af4f8b822d8-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.092733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.053647 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.092952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.078091 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.143331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.143091 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.169681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.169581 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.188337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.188281 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.228401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.213181 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klj49\" (UniqueName: \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-kube-api-access-klj49\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.228401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.219457 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:25:38.250392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.249417 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:38.254609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.250815 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:25:38.396192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.356748 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:25:38.446333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.426524 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:25:38.605741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.595458 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:25:38.618931 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:25:38.618890 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:25:38.652490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.652440 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:25:38.742333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.736865 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:25:38.742333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.737633 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:25:38.751459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:38.751410 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:25:39.243334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.223226 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 06:25:39.243334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.225098 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:25:39.539837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.539809 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:25:39.549934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.549912 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 06:25:39.607030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.606971 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:39.607185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.607136 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:39.607248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.607228 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:39.620632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.618574 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx"] Oct 30 06:25:39.630790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.630767 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:39.634149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.633852 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" event={"ID":"99a380d4-e404-41ed-9e49-1af4f8b822d8","Type":"ContainerStarted","Data":"a430d19edd7a16edd08352a460cd45555df6b61b136ea855684d0771f27446a5"} Oct 30 06:25:39.645069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.644507 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx"] Oct 30 06:25:39.645069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.644572 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766"] Oct 30 06:25:39.646367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.646339 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:25:39.646879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.646863 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:25:39.646991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.646978 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:25:39.647058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.647025 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:25:39.734780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.734742 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbf4\" (UniqueName: \"kubernetes.io/projected/80a35c97-31ea-4ed3-b993-17047dc03835-kube-api-access-qvbf4\") pod \"maas-api-revoke-sub-805327ee-57gzx\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:39.734956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.734867 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805327ee-57gzx\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:39.734956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.734921 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805327ee-57gzx\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:39.836011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.835970 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805327ee-57gzx\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:39.836408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.836050 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805327ee-57gzx\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:39.836408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.836290 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbf4\" (UniqueName: \"kubernetes.io/projected/80a35c97-31ea-4ed3-b993-17047dc03835-kube-api-access-qvbf4\") pod \"maas-api-revoke-sub-805327ee-57gzx\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:39.837705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.837678 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805327ee-57gzx\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:39.841286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.841261 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805327ee-57gzx\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:39.897334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:39.876225 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbf4\" (UniqueName: \"kubernetes.io/projected/80a35c97-31ea-4ed3-b993-17047dc03835-kube-api-access-qvbf4\") pod \"maas-api-revoke-sub-805327ee-57gzx\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:40.014832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.014795 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:40.047196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.047158 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq"] Oct 30 06:25:40.096693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.075759 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.141992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.141883 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq"] Oct 30 06:25:40.249001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.248955 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdm85\" (UniqueName: \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-kube-api-access-pdm85\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.249176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.249090 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.249176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.249149 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc10096-6be7-4413-bb12-684879f00809-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.249411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.249196 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.249411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.249223 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.249411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.249341 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.250036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.250012 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc10096-6be7-4413-bb12-684879f00809-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.250139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.250068 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.250139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.250086 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.346183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.346145 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs"] Oct 30 06:25:40.392688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.392653 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs"] Oct 30 06:25:40.400386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.393028 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.409299 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.409410 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc10096-6be7-4413-bb12-684879f00809-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.409462 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.409479 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.409531 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.409606 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc10096-6be7-4413-bb12-684879f00809-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.409659 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.409674 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.409730 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdm85\" (UniqueName: \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-kube-api-access-pdm85\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.410934 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.413980 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.414151 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.414413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.414321 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.415283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.414799 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc10096-6be7-4413-bb12-684879f00809-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.415283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.414931 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc10096-6be7-4413-bb12-684879f00809-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.415283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.415017 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.418798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.418758 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ec27ed-openshift-default-dockercfg-bv2f6\"" Oct 30 06:25:40.438651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.438622 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.440490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.440464 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdm85\" (UniqueName: \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-kube-api-access-pdm85\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.450645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.449689 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:40.643818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.627672 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-socket\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.643818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.627787 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-data\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.643818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.627850 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-envoy\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.643818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.627927 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-certs\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.643818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.627962 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istiod-ca-cert\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.643818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.628021 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4xm5\" (UniqueName: \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-kube-api-access-w4xm5\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.643818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.628107 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-credential-socket\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.643818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.628142 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-podinfo\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.643818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.628208 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-token\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.701528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.679504 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx"] Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.729986 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-envoy\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.730068 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-certs\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.730100 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istiod-ca-cert\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.730152 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4xm5\" (UniqueName: \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-kube-api-access-w4xm5\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.731108 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-credential-socket\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.731158 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-podinfo\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.731230 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-token\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.731278 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-socket\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.731398 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-data\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.731983 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-data\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.732775 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-credential-socket\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.733011 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-certs\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.733803 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istiod-ca-cert\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.738344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.736075 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-socket\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.742727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.739175 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-envoy\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.751137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.751090 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-podinfo\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.778652 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.775446 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4xm5\" (UniqueName: \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-kube-api-access-w4xm5\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.822299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.812823 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-token\") pod \"e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:40.823282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.823246 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" event={"ID":"99a380d4-e404-41ed-9e49-1af4f8b822d8","Type":"ContainerStarted","Data":"28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93"} Oct 30 06:25:40.915393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:40.915236 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" podStartSLOduration=3.915216935 podStartE2EDuration="3.915216935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:40.903046074 +0000 UTC m=+1954.899003076" watchObservedRunningTime="2026-10-30 06:25:40.915216935 +0000 UTC m=+1954.911173946" Oct 30 06:25:41.122367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.122331 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:41.250122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.250092 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:41.251156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.251054 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 06:25:41.251156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.251103 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 06:25:41.326796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.326766 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq"] Oct 30 06:25:41.338256 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:25:41.338229 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc10096_6be7_4413_bb12_684879f00809.slice/crio-6db7413688324966f7e6f1e948be87601d72a1b1151bc3f023e5841071169aa3 WatchSource:0}: Error finding container 6db7413688324966f7e6f1e948be87601d72a1b1151bc3f023e5841071169aa3: Status 404 returned error can't find the container with id 6db7413688324966f7e6f1e948be87601d72a1b1151bc3f023e5841071169aa3 Oct 30 06:25:41.344326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.342852 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:41.344326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.342944 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:41.344326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.342985 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:41.604180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.600439 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs"] Oct 30 06:25:41.623110 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:25:41.623078 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6a40bf_862a_49e7_9fd5_a75582747c0d.slice/crio-994b70315786465c001e9f34941ebd6c9de2cda820c4e63e7b7d0e0177fc03c9 WatchSource:0}: Error finding container 994b70315786465c001e9f34941ebd6c9de2cda820c4e63e7b7d0e0177fc03c9: Status 404 returned error can't find the container with id 994b70315786465c001e9f34941ebd6c9de2cda820c4e63e7b7d0e0177fc03c9 Oct 30 06:25:41.628694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.628659 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:41.628915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.628898 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:41.629080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.629059 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:41.850371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.849808 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" event={"ID":"80a35c97-31ea-4ed3-b993-17047dc03835","Type":"ContainerStarted","Data":"830010d8f74056dcf3b66e773433e1478f1494b8e50d198edeaec19a961ace4e"} Oct 30 06:25:41.850371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.849872 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" event={"ID":"80a35c97-31ea-4ed3-b993-17047dc03835","Type":"ContainerStarted","Data":"012453f937ad7b90692b218f93f38bc54e78e8ee285c2b01c4fd6a0767e7846f"} Oct 30 06:25:41.863534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.862228 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" event={"ID":"bcc10096-6be7-4413-bb12-684879f00809","Type":"ContainerStarted","Data":"d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca"} Oct 30 06:25:41.863534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.862294 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" event={"ID":"bcc10096-6be7-4413-bb12-684879f00809","Type":"ContainerStarted","Data":"6db7413688324966f7e6f1e948be87601d72a1b1151bc3f023e5841071169aa3"} Oct 30 06:25:41.894332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.883077 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" event={"ID":"6d6a40bf-862a-49e7-9fd5-a75582747c0d","Type":"ContainerStarted","Data":"994b70315786465c001e9f34941ebd6c9de2cda820c4e63e7b7d0e0177fc03c9"} Oct 30 06:25:41.989327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:41.989228 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" podStartSLOduration=2.989206385 podStartE2EDuration="2.989206385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:41.906259554 +0000 UTC m=+1955.902216567" watchObservedRunningTime="2026-10-30 06:25:41.989206385 +0000 UTC m=+1955.985163397" Oct 30 06:25:42.276362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.253175 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 06:25:42.316330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.310189 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 06:25:42.378326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.374644 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" podStartSLOduration=3.374622848 podStartE2EDuration="3.374622848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:41.972658682 +0000 UTC m=+1955.968615692" watchObservedRunningTime="2026-10-30 06:25:42.374622848 +0000 UTC m=+1956.370579859" Oct 30 06:25:42.466414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.466369 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29"] Oct 30 06:25:42.537637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.537603 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:42.540902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.537860 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:42.549365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.547019 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:25:42.549365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.547069 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 06:25:42.549585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.549461 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29"] Oct 30 06:25:42.743754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.743141 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0651b4ba-52z29\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:42.743754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.743234 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0651b4ba-52z29\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:42.743754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.743279 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77s9c\" (UniqueName: \"kubernetes.io/projected/49a3762c-7164-4716-8ab6-a24fe8c6a22b-kube-api-access-77s9c\") pod \"maas-api-revoke-sub-0651b4ba-52z29\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:42.846535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.846436 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0651b4ba-52z29\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:42.850377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.850336 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0651b4ba-52z29\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:42.858714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.858680 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77s9c\" (UniqueName: \"kubernetes.io/projected/49a3762c-7164-4716-8ab6-a24fe8c6a22b-kube-api-access-77s9c\") pod \"maas-api-revoke-sub-0651b4ba-52z29\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:42.924857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.851638 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0651b4ba-52z29\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:42.976393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:42.951429 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77s9c\" (UniqueName: \"kubernetes.io/projected/49a3762c-7164-4716-8ab6-a24fe8c6a22b-kube-api-access-77s9c\") pod \"maas-api-revoke-sub-0651b4ba-52z29\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:43.005387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.003938 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0651b4ba-52z29\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:43.041067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.041024 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" event={"ID":"6d6a40bf-862a-49e7-9fd5-a75582747c0d","Type":"ContainerStarted","Data":"999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a"} Oct 30 06:25:43.047275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.047230 5001 generic.go:356] "Generic (PLEG): container finished" podID="80a35c97-31ea-4ed3-b993-17047dc03835" containerID="830010d8f74056dcf3b66e773433e1478f1494b8e50d198edeaec19a961ace4e" exitCode=0 Oct 30 06:25:43.053767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.047577 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" event={"ID":"80a35c97-31ea-4ed3-b993-17047dc03835","Type":"ContainerDied","Data":"830010d8f74056dcf3b66e773433e1478f1494b8e50d198edeaec19a961ace4e"} Oct 30 06:25:43.132333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.125864 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:43.186334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.161657 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:43.250709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.237114 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" podStartSLOduration=3.237093866 podStartE2EDuration="3.237093866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:43.162060196 +0000 UTC m=+1957.158017207" watchObservedRunningTime="2026-10-30 06:25:43.237093866 +0000 UTC m=+1957.233050877" Oct 30 06:25:43.267958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.267864 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:43.270342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.269467 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 06:25:43.270342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.269528 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 06:25:43.452789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.452752 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:25:43.453036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.453008 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 06:25:43.935835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.869671 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7"] Oct 30 06:25:43.935835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:43.869922 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" containerID="cri-o://4bdd58861e0c87f1c1043205a2593cfeeafeb774ad0079d8c4cdcd78edad54a9" gracePeriod=30 Oct 30 06:25:44.266619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:44.266502 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 06:25:44.266619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:44.266580 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 06:25:44.529559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:44.529458 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:25:44.529840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:44.529809 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 06:25:44.529968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:44.529538 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:44.530135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:44.530123 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:25:44.842676 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:25:44.842636 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a3762c_7164_4716_8ab6_a24fe8c6a22b.slice/crio-8adfb37602f18707d5a39fc78d7868f90c48c608d28f377da0ba98751f79654d WatchSource:0}: Error finding container 8adfb37602f18707d5a39fc78d7868f90c48c608d28f377da0ba98751f79654d: Status 404 returned error can't find the container with id 8adfb37602f18707d5a39fc78d7868f90c48c608d28f377da0ba98751f79654d Oct 30 06:25:44.849605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:44.849575 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29"] Oct 30 06:25:45.114334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.113937 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:45.271330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.251226 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 06:25:45.271330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.251291 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 06:25:45.271330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.251619 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-xks-ca\") pod \"80a35c97-31ea-4ed3-b993-17047dc03835\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " Oct 30 06:25:45.271330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.251674 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-openshift-ca\") pod \"80a35c97-31ea-4ed3-b993-17047dc03835\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " Oct 30 06:25:45.271330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.251758 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a35c97-31ea-4ed3-b993-17047dc03835-kube-api-access-qvbf4\" (UniqueName: \"kubernetes.io/projected/80a35c97-31ea-4ed3-b993-17047dc03835-kube-api-access-qvbf4\") pod \"80a35c97-31ea-4ed3-b993-17047dc03835\" (UID: \"80a35c97-31ea-4ed3-b993-17047dc03835\") " Oct 30 06:25:45.296178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.296108 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-openshift-ca" pod "80a35c97-31ea-4ed3-b993-17047dc03835" (UID: "80a35c97-31ea-4ed3-b993-17047dc03835"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:45.320169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.319520 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-xks-ca" pod "80a35c97-31ea-4ed3-b993-17047dc03835" (UID: "80a35c97-31ea-4ed3-b993-17047dc03835"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:45.320169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.319648 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a35c97-31ea-4ed3-b993-17047dc03835-kube-api-access-qvbf4" pod "80a35c97-31ea-4ed3-b993-17047dc03835" (UID: "80a35c97-31ea-4ed3-b993-17047dc03835"). InnerVolumeSpecName "kube-api-access-qvbf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:45.331924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.331891 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" Oct 30 06:25:45.331924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.331902 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx" event={"ID":"80a35c97-31ea-4ed3-b993-17047dc03835","Type":"ContainerDied","Data":"012453f937ad7b90692b218f93f38bc54e78e8ee285c2b01c4fd6a0767e7846f"} Oct 30 06:25:45.332127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.331956 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="012453f937ad7b90692b218f93f38bc54e78e8ee285c2b01c4fd6a0767e7846f" Oct 30 06:25:45.333745 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.333651 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" event={"ID":"49a3762c-7164-4716-8ab6-a24fe8c6a22b","Type":"ContainerStarted","Data":"8adfb37602f18707d5a39fc78d7868f90c48c608d28f377da0ba98751f79654d"} Oct 30 06:25:45.380330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.379568 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:45.380330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.379604 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80a35c97-31ea-4ed3-b993-17047dc03835-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:45.380330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.379619 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbf4\" (UniqueName: \"kubernetes.io/projected/80a35c97-31ea-4ed3-b993-17047dc03835-kube-api-access-qvbf4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:45.451052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.450965 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:25:45.451052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:45.451008 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 06:25:46.258372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:46.257735 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 06:25:46.258372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:46.257801 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 06:25:46.399332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:46.389298 5001 generic.go:356] "Generic (PLEG): container finished" podID="49a3762c-7164-4716-8ab6-a24fe8c6a22b" containerID="bc6fa23539b045905e60ae2352fa8dd2116265168fa72c1590042aeb1e817940" exitCode=0 Oct 30 06:25:46.399332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:46.390996 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" event={"ID":"49a3762c-7164-4716-8ab6-a24fe8c6a22b","Type":"ContainerDied","Data":"bc6fa23539b045905e60ae2352fa8dd2116265168fa72c1590042aeb1e817940"} Oct 30 06:25:46.450697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:46.450606 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:25:46.450697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:46.450671 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 06:25:47.250938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:47.250899 5001 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 06:25:47.251137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:47.250977 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 06:25:47.462359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:47.461770 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:25:47.462359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:47.461847 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 06:25:47.584331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:47.578432 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:25:47.584331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:47.578521 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:25:48.036332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:47.999374 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4"] Oct 30 06:25:48.036332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.000404 5001 cpu_manager.go:414] "containerMap: removing container" podUID="80a35c97-31ea-4ed3-b993-17047dc03835" containerName="revoke-keys" Oct 30 06:25:48.036332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.000426 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a35c97-31ea-4ed3-b993-17047dc03835" containerName="revoke-keys" Oct 30 06:25:48.036332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.000652 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a35c97-31ea-4ed3-b993-17047dc03835" containerName="revoke-keys" Oct 30 06:25:48.056576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.056548 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4"] Oct 30 06:25:48.069758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.057209 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:48.104781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.104753 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:48.180332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.177269 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-openshift-ca\") pod \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " Oct 30 06:25:48.180332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.177378 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-xks-ca\") pod \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " Oct 30 06:25:48.180332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.177446 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a3762c-7164-4716-8ab6-a24fe8c6a22b-kube-api-access-77s9c\" (UniqueName: \"kubernetes.io/projected/49a3762c-7164-4716-8ab6-a24fe8c6a22b-kube-api-access-77s9c\") pod \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\" (UID: \"49a3762c-7164-4716-8ab6-a24fe8c6a22b\") " Oct 30 06:25:48.180332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.177711 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njdxz\" (UniqueName: \"kubernetes.io/projected/35a45917-f6c7-4b51-b519-113448b07bb9-kube-api-access-njdxz\") pod \"maas-api-revoke-sub-af895423-9ncg4\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:48.180332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.177778 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af895423-9ncg4\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:48.180332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.177858 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af895423-9ncg4\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:48.180332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.178479 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-openshift-ca" pod "49a3762c-7164-4716-8ab6-a24fe8c6a22b" (UID: "49a3762c-7164-4716-8ab6-a24fe8c6a22b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:48.200653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.200600 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-xks-ca" pod "49a3762c-7164-4716-8ab6-a24fe8c6a22b" (UID: "49a3762c-7164-4716-8ab6-a24fe8c6a22b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:48.202881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.202846 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a3762c-7164-4716-8ab6-a24fe8c6a22b-kube-api-access-77s9c" pod "49a3762c-7164-4716-8ab6-a24fe8c6a22b" (UID: "49a3762c-7164-4716-8ab6-a24fe8c6a22b"). InnerVolumeSpecName "kube-api-access-77s9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:48.291243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.289210 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njdxz\" (UniqueName: \"kubernetes.io/projected/35a45917-f6c7-4b51-b519-113448b07bb9-kube-api-access-njdxz\") pod \"maas-api-revoke-sub-af895423-9ncg4\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:48.291243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.289296 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af895423-9ncg4\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:48.291243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.289398 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af895423-9ncg4\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:48.291243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.289562 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:48.291243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.289581 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a3762c-7164-4716-8ab6-a24fe8c6a22b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:48.291243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.289595 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77s9c\" (UniqueName: \"kubernetes.io/projected/49a3762c-7164-4716-8ab6-a24fe8c6a22b-kube-api-access-77s9c\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:48.291243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.291134 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af895423-9ncg4\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:48.293084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.293049 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af895423-9ncg4\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:48.321890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.321823 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njdxz\" (UniqueName: \"kubernetes.io/projected/35a45917-f6c7-4b51-b519-113448b07bb9-kube-api-access-njdxz\") pod \"maas-api-revoke-sub-af895423-9ncg4\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:48.416383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.415057 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:48.430208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.430175 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" event={"ID":"49a3762c-7164-4716-8ab6-a24fe8c6a22b","Type":"ContainerDied","Data":"8adfb37602f18707d5a39fc78d7868f90c48c608d28f377da0ba98751f79654d"} Oct 30 06:25:48.430348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.430238 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8adfb37602f18707d5a39fc78d7868f90c48c608d28f377da0ba98751f79654d" Oct 30 06:25:48.430348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.430326 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29" Oct 30 06:25:48.451087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.451033 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:25:48.451219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.451093 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 06:25:48.908346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.908285 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:48.909967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.909938 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:48.911753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:48.911696 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:25:49.105475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.105442 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4"] Oct 30 06:25:49.108922 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:25:49.108888 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a45917_f6c7_4b51_b519_113448b07bb9.slice/crio-5f802e07a63ab0234f2fab35b097373c44f654544fb4cfc0fac19eaf86c51235 WatchSource:0}: Error finding container 5f802e07a63ab0234f2fab35b097373c44f654544fb4cfc0fac19eaf86c51235: Status 404 returned error can't find the container with id 5f802e07a63ab0234f2fab35b097373c44f654544fb4cfc0fac19eaf86c51235 Oct 30 06:25:49.442229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.442188 5001 generic.go:356] "Generic (PLEG): container finished" podID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerID="4bdd58861e0c87f1c1043205a2593cfeeafeb774ad0079d8c4cdcd78edad54a9" exitCode=0 Oct 30 06:25:49.442463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.442231 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" event={"ID":"32cdd044-ecb6-485e-9b34-9d1362aa3bfa","Type":"ContainerDied","Data":"4bdd58861e0c87f1c1043205a2593cfeeafeb774ad0079d8c4cdcd78edad54a9"} Oct 30 06:25:49.443990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.443965 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" event={"ID":"35a45917-f6c7-4b51-b519-113448b07bb9","Type":"ContainerStarted","Data":"5f802e07a63ab0234f2fab35b097373c44f654544fb4cfc0fac19eaf86c51235"} Oct 30 06:25:49.450684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.450659 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:25:49.450820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.450712 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 06:25:49.867350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.864601 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:25:49.961961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.961915 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-kube-api-access-xrbf5\" (UniqueName: \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-kube-api-access-xrbf5\") pod \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " Oct 30 06:25:49.962479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.962021 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-data\") pod \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " Oct 30 06:25:49.962479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.962096 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-podinfo\") pod \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " Oct 30 06:25:49.962479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.962162 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-envoy\") pod \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " Oct 30 06:25:49.962479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.962216 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-token\" (UniqueName: \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-token\") pod \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " Oct 30 06:25:49.962479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.962268 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istiod-ca-cert\") pod \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " Oct 30 06:25:49.962479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.962366 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-certs\") pod \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " Oct 30 06:25:49.962479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.962418 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-socket\") pod \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " Oct 30 06:25:49.962479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.962472 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-credential-socket\") pod \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\" (UID: \"32cdd044-ecb6-485e-9b34-9d1362aa3bfa\") " Oct 30 06:25:49.969347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.969290 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-credential-socket" pod "32cdd044-ecb6-485e-9b34-9d1362aa3bfa" (UID: "32cdd044-ecb6-485e-9b34-9d1362aa3bfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:49.969688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.969659 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-certs" pod "32cdd044-ecb6-485e-9b34-9d1362aa3bfa" (UID: "32cdd044-ecb6-485e-9b34-9d1362aa3bfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:49.970228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.969999 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istiod-ca-cert" pod "32cdd044-ecb6-485e-9b34-9d1362aa3bfa" (UID: "32cdd044-ecb6-485e-9b34-9d1362aa3bfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:49.970674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.970391 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-socket" pod "32cdd044-ecb6-485e-9b34-9d1362aa3bfa" (UID: "32cdd044-ecb6-485e-9b34-9d1362aa3bfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:49.971475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.971453 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-data" pod "32cdd044-ecb6-485e-9b34-9d1362aa3bfa" (UID: "32cdd044-ecb6-485e-9b34-9d1362aa3bfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:49.976111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.976075 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-envoy" pod "32cdd044-ecb6-485e-9b34-9d1362aa3bfa" (UID: "32cdd044-ecb6-485e-9b34-9d1362aa3bfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:49.998322 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.985919 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-podinfo" pod "32cdd044-ecb6-485e-9b34-9d1362aa3bfa" (UID: "32cdd044-ecb6-485e-9b34-9d1362aa3bfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:25:49.998322 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.995944 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-kube-api-access-xrbf5" pod "32cdd044-ecb6-485e-9b34-9d1362aa3bfa" (UID: "32cdd044-ecb6-485e-9b34-9d1362aa3bfa"). InnerVolumeSpecName "kube-api-access-xrbf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:49.998322 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:49.996649 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-token" pod "32cdd044-ecb6-485e-9b34-9d1362aa3bfa" (UID: "32cdd044-ecb6-485e-9b34-9d1362aa3bfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:50.064023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.063985 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrbf5\" (UniqueName: \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-kube-api-access-xrbf5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:50.064023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.064022 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:50.064250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.064036 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:50.064250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.064047 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:50.064250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.064057 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:50.064250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.064068 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:50.064250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.064078 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:50.064250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.064088 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:50.064250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.064100 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cdd044-ecb6-485e-9b34-9d1362aa3bfa-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:50.455351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.455286 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:50.456041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.456016 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:50.456960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.456942 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:25:50.457573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.457547 5001 generic.go:356] "Generic (PLEG): container finished" podID="35a45917-f6c7-4b51-b519-113448b07bb9" containerID="ef44f7c5f11f7eb5b9fb637924b96e72db0211fdb9aea5504f7576d2b046c9ac" exitCode=0 Oct 30 06:25:50.457698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.457599 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" event={"ID":"35a45917-f6c7-4b51-b519-113448b07bb9","Type":"ContainerDied","Data":"ef44f7c5f11f7eb5b9fb637924b96e72db0211fdb9aea5504f7576d2b046c9ac"} Oct 30 06:25:50.459507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.459485 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" Oct 30 06:25:50.459616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.459522 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7" event={"ID":"32cdd044-ecb6-485e-9b34-9d1362aa3bfa","Type":"ContainerDied","Data":"74770bff3864ccd847e10ad6beb3d9ff873af2fa962265ab9e93fcf1052b8fa7"} Oct 30 06:25:50.459616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.459586 5001 scope.go:122] "RemoveContainer" containerID="4bdd58861e0c87f1c1043205a2593cfeeafeb774ad0079d8c4cdcd78edad54a9" Oct 30 06:25:50.706521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.706478 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7"] Oct 30 06:25:50.714838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:50.714802 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2ltk7"] Oct 30 06:25:51.469331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.469243 5001 generic.go:356] "Generic (PLEG): container finished" podID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerID="e1d702587347d396cebe122245345bb5a55542840cc54958553a06665dff9941" exitCode=0 Oct 30 06:25:51.469796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.469334 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" event={"ID":"03ed5ca9-d682-4636-a96b-e7d2b84b8455","Type":"ContainerDied","Data":"e1d702587347d396cebe122245345bb5a55542840cc54958553a06665dff9941"} Oct 30 06:25:51.469872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.469823 5001 scope.go:122] "RemoveContainer" containerID="e1d702587347d396cebe122245345bb5a55542840cc54958553a06665dff9941" Oct 30 06:25:51.471697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.471664 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:51.471810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.471738 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:51.471810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.471770 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:51.684345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.684294 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:51.787934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.787811 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a45917-f6c7-4b51-b519-113448b07bb9-kube-api-access-njdxz\" (UniqueName: \"kubernetes.io/projected/35a45917-f6c7-4b51-b519-113448b07bb9-kube-api-access-njdxz\") pod \"35a45917-f6c7-4b51-b519-113448b07bb9\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " Oct 30 06:25:51.787934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.787881 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-xks-ca\") pod \"35a45917-f6c7-4b51-b519-113448b07bb9\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " Oct 30 06:25:51.788500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.788076 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-openshift-ca\") pod \"35a45917-f6c7-4b51-b519-113448b07bb9\" (UID: \"35a45917-f6c7-4b51-b519-113448b07bb9\") " Oct 30 06:25:51.789174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.789141 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-openshift-ca" pod "35a45917-f6c7-4b51-b519-113448b07bb9" (UID: "35a45917-f6c7-4b51-b519-113448b07bb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:51.797173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.797144 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-xks-ca" pod "35a45917-f6c7-4b51-b519-113448b07bb9" (UID: "35a45917-f6c7-4b51-b519-113448b07bb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:51.798834 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.798804 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a45917-f6c7-4b51-b519-113448b07bb9-kube-api-access-njdxz" pod "35a45917-f6c7-4b51-b519-113448b07bb9" (UID: "35a45917-f6c7-4b51-b519-113448b07bb9"). InnerVolumeSpecName "kube-api-access-njdxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:51.890658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.890617 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njdxz\" (UniqueName: \"kubernetes.io/projected/35a45917-f6c7-4b51-b519-113448b07bb9-kube-api-access-njdxz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:51.890658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.890650 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:51.890658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:51.890661 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a45917-f6c7-4b51-b519-113448b07bb9-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:25:52.117736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:52.117703 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" path="/var/lib/kubelet/pods/32cdd044-ecb6-485e-9b34-9d1362aa3bfa/volumes" Oct 30 06:25:52.523846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:52.523752 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" Oct 30 06:25:52.523846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:52.523777 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4" event={"ID":"35a45917-f6c7-4b51-b519-113448b07bb9","Type":"ContainerDied","Data":"5f802e07a63ab0234f2fab35b097373c44f654544fb4cfc0fac19eaf86c51235"} Oct 30 06:25:52.524447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:52.523857 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f802e07a63ab0234f2fab35b097373c44f654544fb4cfc0fac19eaf86c51235" Oct 30 06:25:52.528568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:52.528539 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" event={"ID":"03ed5ca9-d682-4636-a96b-e7d2b84b8455","Type":"ContainerStarted","Data":"99a957f6bdca6072b90adf2cc6a837466a8fb65f60154d49667b7449a7a90af3"} Oct 30 06:25:53.515026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:53.514983 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:25:53.515363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:53.515338 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:25:53.515445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:53.515392 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 06:25:54.515668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:54.515630 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:25:54.516201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:54.515689 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 06:25:55.515005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:55.514969 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:25:55.515200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:55.515040 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 06:25:56.515149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:56.515106 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:25:56.515589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:56.515180 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 06:25:57.515520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:57.515476 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:25:57.515977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:57.515563 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 30 06:25:58.323184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.323151 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv"] Oct 30 06:25:58.323790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.323776 5001 cpu_manager.go:414] "containerMap: removing container" podUID="49a3762c-7164-4716-8ab6-a24fe8c6a22b" containerName="revoke-keys" Oct 30 06:25:58.323847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.323792 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a3762c-7164-4716-8ab6-a24fe8c6a22b" containerName="revoke-keys" Oct 30 06:25:58.323847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.323815 5001 cpu_manager.go:414] "containerMap: removing container" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" Oct 30 06:25:58.323847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.323820 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" Oct 30 06:25:58.323847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.323836 5001 cpu_manager.go:414] "containerMap: removing container" podUID="35a45917-f6c7-4b51-b519-113448b07bb9" containerName="revoke-keys" Oct 30 06:25:58.323847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.323841 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a45917-f6c7-4b51-b519-113448b07bb9" containerName="revoke-keys" Oct 30 06:25:58.324025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.323942 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cdd044-ecb6-485e-9b34-9d1362aa3bfa" containerName="istio-proxy" Oct 30 06:25:58.324025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.323953 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a45917-f6c7-4b51-b519-113448b07bb9" containerName="revoke-keys" Oct 30 06:25:58.324025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.323964 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a3762c-7164-4716-8ab6-a24fe8c6a22b" containerName="revoke-keys" Oct 30 06:25:58.327609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.327579 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:25:58.347547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.347501 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:25:58.351782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.351748 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv"] Oct 30 06:25:58.366566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.366537 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:25:58.367507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.366535 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:25:58.367828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.367807 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:25:58.429002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.428961 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8shw\" (UniqueName: \"kubernetes.io/projected/2b085c0c-26dd-40b6-b038-49483722314c-kube-api-access-b8shw\") pod \"maas-api-revoke-sub-842c8d2b-xxxfv\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:25:58.429173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.429100 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-842c8d2b-xxxfv\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:25:58.429249 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.429233 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-842c8d2b-xxxfv\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:25:58.521174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.521144 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:25:58.521706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.521565 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:25:58.522239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.522220 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:25:58.530927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.530903 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-842c8d2b-xxxfv\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:25:58.531046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.531026 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-842c8d2b-xxxfv\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:25:58.531130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.531118 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8shw\" (UniqueName: \"kubernetes.io/projected/2b085c0c-26dd-40b6-b038-49483722314c-kube-api-access-b8shw\") pod \"maas-api-revoke-sub-842c8d2b-xxxfv\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:25:58.531672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.531622 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-842c8d2b-xxxfv\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:25:58.533406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.533386 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-842c8d2b-xxxfv\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:25:58.561959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.561934 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8shw\" (UniqueName: \"kubernetes.io/projected/2b085c0c-26dd-40b6-b038-49483722314c-kube-api-access-b8shw\") pod \"maas-api-revoke-sub-842c8d2b-xxxfv\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:25:58.671864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:58.645713 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:25:59.049440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:59.049411 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv"] Oct 30 06:25:59.050535 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:25:59.050500 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b085c0c_26dd_40b6_b038_49483722314c.slice/crio-58c3ab3112d709cee02168dea1ad9b2c99fd02aafd9632b049cde5008583cc92 WatchSource:0}: Error finding container 58c3ab3112d709cee02168dea1ad9b2c99fd02aafd9632b049cde5008583cc92: Status 404 returned error can't find the container with id 58c3ab3112d709cee02168dea1ad9b2c99fd02aafd9632b049cde5008583cc92 Oct 30 06:25:59.619230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:59.619126 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" event={"ID":"2b085c0c-26dd-40b6-b038-49483722314c","Type":"ContainerStarted","Data":"432bf344ef1df24dacfa6600977ce5d18aded6475cb49baf850be816f291fffb"} Oct 30 06:25:59.619230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:59.619192 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" event={"ID":"2b085c0c-26dd-40b6-b038-49483722314c","Type":"ContainerStarted","Data":"58c3ab3112d709cee02168dea1ad9b2c99fd02aafd9632b049cde5008583cc92"} Oct 30 06:25:59.662015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:25:59.661952 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" podStartSLOduration=1.661934385 podStartE2EDuration="1.661934385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:59.659152131 +0000 UTC m=+1973.655109142" watchObservedRunningTime="2026-10-30 06:25:59.661934385 +0000 UTC m=+1973.657891400" Oct 30 06:26:00.486207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.486162 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr"] Oct 30 06:26:00.491581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.491558 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:00.508973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.508941 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr"] Oct 30 06:26:00.565068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.565035 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c6c9c4e-99pcr\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:00.565283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.565096 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz7hn\" (UniqueName: \"kubernetes.io/projected/45867f92-1f2e-4c09-9c71-3ed44574a01d-kube-api-access-rz7hn\") pod \"maas-api-revoke-sub-8c6c9c4e-99pcr\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:00.565283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.565237 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c6c9c4e-99pcr\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:00.628933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.628898 5001 generic.go:356] "Generic (PLEG): container finished" podID="2b085c0c-26dd-40b6-b038-49483722314c" containerID="432bf344ef1df24dacfa6600977ce5d18aded6475cb49baf850be816f291fffb" exitCode=0 Oct 30 06:26:00.629374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.628983 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" event={"ID":"2b085c0c-26dd-40b6-b038-49483722314c","Type":"ContainerDied","Data":"432bf344ef1df24dacfa6600977ce5d18aded6475cb49baf850be816f291fffb"} Oct 30 06:26:00.666532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.666499 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c6c9c4e-99pcr\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:00.666760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.666728 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c6c9c4e-99pcr\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:00.666842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.666787 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz7hn\" (UniqueName: \"kubernetes.io/projected/45867f92-1f2e-4c09-9c71-3ed44574a01d-kube-api-access-rz7hn\") pod \"maas-api-revoke-sub-8c6c9c4e-99pcr\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:00.667520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.667490 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c6c9c4e-99pcr\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:00.670408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.670379 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c6c9c4e-99pcr\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:00.690651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.690620 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz7hn\" (UniqueName: \"kubernetes.io/projected/45867f92-1f2e-4c09-9c71-3ed44574a01d-kube-api-access-rz7hn\") pod \"maas-api-revoke-sub-8c6c9c4e-99pcr\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:00.808076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:00.807990 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:01.121375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:01.121334 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr"] Oct 30 06:26:01.123246 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:01.123222 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45867f92_1f2e_4c09_9c71_3ed44574a01d.slice/crio-7ca00269ccd26584fd40a648501d63509c3bd58f5f6b671477c6e0ee5a259e71 WatchSource:0}: Error finding container 7ca00269ccd26584fd40a648501d63509c3bd58f5f6b671477c6e0ee5a259e71: Status 404 returned error can't find the container with id 7ca00269ccd26584fd40a648501d63509c3bd58f5f6b671477c6e0ee5a259e71 Oct 30 06:26:01.640721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:01.640680 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" event={"ID":"45867f92-1f2e-4c09-9c71-3ed44574a01d","Type":"ContainerStarted","Data":"690a07e627efef9b1a6b24fb0fde02898d9667fa0ebd1a29a0bbcb06394a3987"} Oct 30 06:26:01.641243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:01.640750 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" event={"ID":"45867f92-1f2e-4c09-9c71-3ed44574a01d","Type":"ContainerStarted","Data":"7ca00269ccd26584fd40a648501d63509c3bd58f5f6b671477c6e0ee5a259e71"} Oct 30 06:26:02.004847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.003823 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" podStartSLOduration=2.00380222 podStartE2EDuration="2.00380222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:01.733937411 +0000 UTC m=+1975.729894422" watchObservedRunningTime="2026-10-30 06:26:02.00380222 +0000 UTC m=+1975.999759230" Oct 30 06:26:02.007818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.007789 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26"] Oct 30 06:26:02.024349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.023467 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26"] Oct 30 06:26:02.024349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.023612 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.041945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.041881 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-adcdcb-openshift-default-dockercfg-lzw5s\"" Oct 30 06:26:02.059514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.059476 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-data\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.059514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.059525 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxw4b\" (UniqueName: \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-kube-api-access-nxw4b\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.059770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.059578 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-credential-socket\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.059770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.059675 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-token\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.059770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.059722 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-certs\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.059770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.059748 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-podinfo\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.060019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.059775 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad31106-b066-48b3-9bc4-a8e442cb719e-istiod-ca-cert\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.060019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.059818 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-envoy\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.060132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.060023 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-socket\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.163848 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-socket\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.164040 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-data\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.164068 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxw4b\" (UniqueName: \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-kube-api-access-nxw4b\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.164105 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-credential-socket\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.164194 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-token\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.164239 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-certs\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.164264 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-podinfo\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.164292 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad31106-b066-48b3-9bc4-a8e442cb719e-istiod-ca-cert\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.164339 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-envoy\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.164990 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-certs\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.166275 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad31106-b066-48b3-9bc4-a8e442cb719e-istiod-ca-cert\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.167222 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-socket\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.168227 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-credential-socket\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.182722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.177784 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-podinfo\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.194756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.188093 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-data\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.197354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.196149 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-envoy\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.205229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.205196 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxw4b\" (UniqueName: \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-kube-api-access-nxw4b\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.208921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.208893 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-token\") pod \"e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.241772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.241747 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:26:02.265613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.265582 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-openshift-ca\") pod \"2b085c0c-26dd-40b6-b038-49483722314c\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " Oct 30 06:26:02.265809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.265654 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-xks-ca\") pod \"2b085c0c-26dd-40b6-b038-49483722314c\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " Oct 30 06:26:02.265809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.265777 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b085c0c-26dd-40b6-b038-49483722314c-kube-api-access-b8shw\" (UniqueName: \"kubernetes.io/projected/2b085c0c-26dd-40b6-b038-49483722314c-kube-api-access-b8shw\") pod \"2b085c0c-26dd-40b6-b038-49483722314c\" (UID: \"2b085c0c-26dd-40b6-b038-49483722314c\") " Oct 30 06:26:02.266023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.265998 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-openshift-ca" pod "2b085c0c-26dd-40b6-b038-49483722314c" (UID: "2b085c0c-26dd-40b6-b038-49483722314c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:02.266279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.266259 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:02.268028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.268005 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-xks-ca" pod "2b085c0c-26dd-40b6-b038-49483722314c" (UID: "2b085c0c-26dd-40b6-b038-49483722314c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:02.268130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.268111 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b085c0c-26dd-40b6-b038-49483722314c-kube-api-access-b8shw" pod "2b085c0c-26dd-40b6-b038-49483722314c" (UID: "2b085c0c-26dd-40b6-b038-49483722314c"). InnerVolumeSpecName "kube-api-access-b8shw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:02.343177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.343146 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:02.367897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.367835 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b085c0c-26dd-40b6-b038-49483722314c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:02.367897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.367868 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8shw\" (UniqueName: \"kubernetes.io/projected/2b085c0c-26dd-40b6-b038-49483722314c-kube-api-access-b8shw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:02.652058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.651971 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" Oct 30 06:26:02.652679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.652619 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv" event={"ID":"2b085c0c-26dd-40b6-b038-49483722314c","Type":"ContainerDied","Data":"58c3ab3112d709cee02168dea1ad9b2c99fd02aafd9632b049cde5008583cc92"} Oct 30 06:26:02.652846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.652830 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58c3ab3112d709cee02168dea1ad9b2c99fd02aafd9632b049cde5008583cc92" Oct 30 06:26:02.654699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.654225 5001 generic.go:356] "Generic (PLEG): container finished" podID="45867f92-1f2e-4c09-9c71-3ed44574a01d" containerID="690a07e627efef9b1a6b24fb0fde02898d9667fa0ebd1a29a0bbcb06394a3987" exitCode=0 Oct 30 06:26:02.654699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.654286 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" event={"ID":"45867f92-1f2e-4c09-9c71-3ed44574a01d","Type":"ContainerDied","Data":"690a07e627efef9b1a6b24fb0fde02898d9667fa0ebd1a29a0bbcb06394a3987"} Oct 30 06:26:02.678684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.678657 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26"] Oct 30 06:26:02.681016 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:02.680991 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad31106_b066_48b3_9bc4_a8e442cb719e.slice/crio-ec64112b1c69fa87d32cf01336736f2089f696598bbb7b0bde396f8aae185bdd WatchSource:0}: Error finding container ec64112b1c69fa87d32cf01336736f2089f696598bbb7b0bde396f8aae185bdd: Status 404 returned error can't find the container with id ec64112b1c69fa87d32cf01336736f2089f696598bbb7b0bde396f8aae185bdd Oct 30 06:26:02.683822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.683787 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:02.683958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.683875 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:02.683958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:02.683916 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:03.084638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:03.084539 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:26:03.666028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:03.665988 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" event={"ID":"4ad31106-b066-48b3-9bc4-a8e442cb719e","Type":"ContainerStarted","Data":"7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c"} Oct 30 06:26:03.666586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:03.666059 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" event={"ID":"4ad31106-b066-48b3-9bc4-a8e442cb719e","Type":"ContainerStarted","Data":"ec64112b1c69fa87d32cf01336736f2089f696598bbb7b0bde396f8aae185bdd"} Oct 30 06:26:03.740434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:03.740380 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" podStartSLOduration=2.740364026 podStartE2EDuration="2.740364026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:03.723384854 +0000 UTC m=+1977.719341865" watchObservedRunningTime="2026-10-30 06:26:03.740364026 +0000 UTC m=+1977.736321103" Oct 30 06:26:03.863862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:03.863834 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:03.940903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:03.935344 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45867f92-1f2e-4c09-9c71-3ed44574a01d-kube-api-access-rz7hn\" (UniqueName: \"kubernetes.io/projected/45867f92-1f2e-4c09-9c71-3ed44574a01d-kube-api-access-rz7hn\") pod \"45867f92-1f2e-4c09-9c71-3ed44574a01d\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " Oct 30 06:26:03.940903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:03.935416 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-xks-ca\") pod \"45867f92-1f2e-4c09-9c71-3ed44574a01d\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " Oct 30 06:26:03.940903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:03.935674 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-openshift-ca\") pod \"45867f92-1f2e-4c09-9c71-3ed44574a01d\" (UID: \"45867f92-1f2e-4c09-9c71-3ed44574a01d\") " Oct 30 06:26:03.940903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:03.936782 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-openshift-ca" pod "45867f92-1f2e-4c09-9c71-3ed44574a01d" (UID: "45867f92-1f2e-4c09-9c71-3ed44574a01d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:03.941995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:03.941398 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45867f92-1f2e-4c09-9c71-3ed44574a01d-kube-api-access-rz7hn" pod "45867f92-1f2e-4c09-9c71-3ed44574a01d" (UID: "45867f92-1f2e-4c09-9c71-3ed44574a01d"). InnerVolumeSpecName "kube-api-access-rz7hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:03.941995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:03.941721 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-xks-ca" pod "45867f92-1f2e-4c09-9c71-3ed44574a01d" (UID: "45867f92-1f2e-4c09-9c71-3ed44574a01d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:04.046331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:04.040582 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:04.046331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:04.040625 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz7hn\" (UniqueName: \"kubernetes.io/projected/45867f92-1f2e-4c09-9c71-3ed44574a01d-kube-api-access-rz7hn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:04.046331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:04.040642 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45867f92-1f2e-4c09-9c71-3ed44574a01d-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:04.363420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:04.353012 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:04.389383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:04.374474 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:04.725400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:04.724694 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" Oct 30 06:26:04.741844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:04.740564 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr" event={"ID":"45867f92-1f2e-4c09-9c71-3ed44574a01d","Type":"ContainerDied","Data":"7ca00269ccd26584fd40a648501d63509c3bd58f5f6b671477c6e0ee5a259e71"} Oct 30 06:26:04.741844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:04.740637 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ca00269ccd26584fd40a648501d63509c3bd58f5f6b671477c6e0ee5a259e71" Oct 30 06:26:04.741844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:04.740665 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:04.764346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:04.754644 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:26:06.254990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:06.253384 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 06:26:06.418561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:06.418512 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:06.500345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:06.499483 5001 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:06.500345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:06.499552 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:06.546393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:06.542470 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:06.586058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:06.586000 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:06.653345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:06.649429 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:06.757685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:06.757641 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:07.264467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:07.264421 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 06:26:07.265814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:07.265786 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:26:07.582852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:07.582767 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:26:08.214283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.214243 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9"] Oct 30 06:26:08.214921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.214904 5001 cpu_manager.go:414] "containerMap: removing container" podUID="45867f92-1f2e-4c09-9c71-3ed44574a01d" containerName="revoke-keys" Oct 30 06:26:08.214990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.214922 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45867f92-1f2e-4c09-9c71-3ed44574a01d" containerName="revoke-keys" Oct 30 06:26:08.214990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.214949 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2b085c0c-26dd-40b6-b038-49483722314c" containerName="revoke-keys" Oct 30 06:26:08.214990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.214954 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b085c0c-26dd-40b6-b038-49483722314c" containerName="revoke-keys" Oct 30 06:26:08.215104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.215074 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="45867f92-1f2e-4c09-9c71-3ed44574a01d" containerName="revoke-keys" Oct 30 06:26:08.215104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.215085 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b085c0c-26dd-40b6-b038-49483722314c" containerName="revoke-keys" Oct 30 06:26:08.220046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.220028 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 06:26:08.241846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.241819 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9"] Oct 30 06:26:08.244577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.244556 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-adcdcb\"" Oct 30 06:26:08.277258 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:26:08.277219 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:26:08.292668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.292287 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52ksq\" (UniqueName: \"kubernetes.io/projected/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-kube-api-access-52ksq\") pod \"payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9\" (UID: \"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 06:26:08.292668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.292495 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9\" (UID: \"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 06:26:08.329624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.329593 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx"] Oct 30 06:26:08.334292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.334266 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 06:26:08.353592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.353552 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-adcdcb-dockercfg-l2hbb\"" Oct 30 06:26:08.417328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.410380 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx"] Oct 30 06:26:08.472469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.467635 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfn5q\" (UniqueName: \"kubernetes.io/projected/e80eaafa-a522-400a-802f-0a54e846cc3d-kube-api-access-rfn5q\") pod \"payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx\" (UID: \"e80eaafa-a522-400a-802f-0a54e846cc3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 06:26:08.472469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.467733 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52ksq\" (UniqueName: \"kubernetes.io/projected/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-kube-api-access-52ksq\") pod \"payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9\" (UID: \"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 06:26:08.472469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.467783 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80eaafa-a522-400a-802f-0a54e846cc3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx\" (UID: \"e80eaafa-a522-400a-802f-0a54e846cc3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 06:26:08.472469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.468028 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9\" (UID: \"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 06:26:08.472469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.469108 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9\" (UID: \"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 06:26:08.567453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.534354 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52ksq\" (UniqueName: \"kubernetes.io/projected/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-kube-api-access-52ksq\") pod \"payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9\" (UID: \"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 06:26:08.567453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.534675 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 06:26:08.724618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.687799 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfn5q\" (UniqueName: \"kubernetes.io/projected/e80eaafa-a522-400a-802f-0a54e846cc3d-kube-api-access-rfn5q\") pod \"payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx\" (UID: \"e80eaafa-a522-400a-802f-0a54e846cc3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 06:26:08.724618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.687897 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80eaafa-a522-400a-802f-0a54e846cc3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx\" (UID: \"e80eaafa-a522-400a-802f-0a54e846cc3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 06:26:08.724618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.689012 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80eaafa-a522-400a-802f-0a54e846cc3d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx\" (UID: \"e80eaafa-a522-400a-802f-0a54e846cc3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 06:26:08.724618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.715862 5001 scope.go:122] "RemoveContainer" containerID="7322dbbe577f4e2915e5d7bdbb8fa682bfc668d838337ce369f49e5a852a3d33" Oct 30 06:26:08.738077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.738018 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfn5q\" (UniqueName: \"kubernetes.io/projected/e80eaafa-a522-400a-802f-0a54e846cc3d-kube-api-access-rfn5q\") pod \"payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx\" (UID: \"e80eaafa-a522-400a-802f-0a54e846cc3d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 06:26:08.754582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.754477 5001 scope.go:122] "RemoveContainer" containerID="104f43372245e7c9331ba428e490609d4baae9b72e9b8f6fc6cb899867c65360" Oct 30 06:26:08.772095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.772059 5001 scope.go:122] "RemoveContainer" containerID="5bdec4624a3b619ea4668eebc4168d755fa53be2e7c039d66850819fa17b8766" Oct 30 06:26:08.788369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.787073 5001 scope.go:122] "RemoveContainer" containerID="af91c76d7597b8ebf3303e37f1617a1827e7a5666cffd5a8e02c87fec30961b3" Oct 30 06:26:08.941330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.941280 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7"] Oct 30 06:26:08.947744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.947723 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:08.950127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.950108 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 06:26:08.973625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.973440 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:26:08.973625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.973512 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:26:08.974215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.974112 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:26:08.974215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.974186 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:26:08.979810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.979769 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7"] Oct 30 06:26:08.997148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.997117 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36eb8991-sn2g7\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:08.997330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.997173 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36eb8991-sn2g7\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:08.997427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:08.997399 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnq75\" (UniqueName: \"kubernetes.io/projected/22b76b59-27df-4697-9670-de78c1cc0c08-kube-api-access-nnq75\") pod \"maas-api-revoke-sub-36eb8991-sn2g7\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:09.045208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.045181 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9"] Oct 30 06:26:09.049881 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:09.049852 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab0ea91_b217_4fc4_9ed9_0c42fc48bfe4.slice/crio-7c5613c3be1690d156569c072b286ef35d2d458e5a86694327ceb970b083915e WatchSource:0}: Error finding container 7c5613c3be1690d156569c072b286ef35d2d458e5a86694327ceb970b083915e: Status 404 returned error can't find the container with id 7c5613c3be1690d156569c072b286ef35d2d458e5a86694327ceb970b083915e Oct 30 06:26:09.099074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.099044 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnq75\" (UniqueName: \"kubernetes.io/projected/22b76b59-27df-4697-9670-de78c1cc0c08-kube-api-access-nnq75\") pod \"maas-api-revoke-sub-36eb8991-sn2g7\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:09.099240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.099204 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36eb8991-sn2g7\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:09.099334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.099256 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36eb8991-sn2g7\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:09.099901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.099876 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36eb8991-sn2g7\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:09.101808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.101790 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36eb8991-sn2g7\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:09.124270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.124250 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnq75\" (UniqueName: \"kubernetes.io/projected/22b76b59-27df-4697-9670-de78c1cc0c08-kube-api-access-nnq75\") pod \"maas-api-revoke-sub-36eb8991-sn2g7\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:09.264334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.264225 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:09.293023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.292986 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx"] Oct 30 06:26:09.593606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.593579 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7"] Oct 30 06:26:09.596842 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:09.596808 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b76b59_27df_4697_9670_de78c1cc0c08.slice/crio-2b3f51290ea5f4f911efd3e410555953d4000443f9d21f0e95c84c2adcf4a9e2 WatchSource:0}: Error finding container 2b3f51290ea5f4f911efd3e410555953d4000443f9d21f0e95c84c2adcf4a9e2: Status 404 returned error can't find the container with id 2b3f51290ea5f4f911efd3e410555953d4000443f9d21f0e95c84c2adcf4a9e2 Oct 30 06:26:09.930635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.930593 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" event={"ID":"e80eaafa-a522-400a-802f-0a54e846cc3d","Type":"ContainerStarted","Data":"f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb"} Oct 30 06:26:09.930838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.930664 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 06:26:09.930838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.930684 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" event={"ID":"e80eaafa-a522-400a-802f-0a54e846cc3d","Type":"ContainerStarted","Data":"a9516232ffc25515ab4d8f43f09ed64f32fe8daee2165c4ec303559d4b4cf0c9"} Oct 30 06:26:09.932366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.932335 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" event={"ID":"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4","Type":"ContainerStarted","Data":"21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e"} Oct 30 06:26:09.932499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.932386 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" event={"ID":"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4","Type":"ContainerStarted","Data":"7c5613c3be1690d156569c072b286ef35d2d458e5a86694327ceb970b083915e"} Oct 30 06:26:09.932499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.932416 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 06:26:09.933674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.933652 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" event={"ID":"22b76b59-27df-4697-9670-de78c1cc0c08","Type":"ContainerStarted","Data":"2b3f51290ea5f4f911efd3e410555953d4000443f9d21f0e95c84c2adcf4a9e2"} Oct 30 06:26:09.973876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:09.973830 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" podStartSLOduration=1.973815587 podStartE2EDuration="1.973815587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:09.972827894 +0000 UTC m=+1983.968784905" watchObservedRunningTime="2026-10-30 06:26:09.973815587 +0000 UTC m=+1983.969772645" Oct 30 06:26:10.019984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:10.019931 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" podStartSLOduration=2.019917466 podStartE2EDuration="2.019917466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:10.014512083 +0000 UTC m=+1984.010469093" watchObservedRunningTime="2026-10-30 06:26:10.019917466 +0000 UTC m=+1984.015874471" Oct 30 06:26:10.943106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:10.943020 5001 generic.go:356] "Generic (PLEG): container finished" podID="22b76b59-27df-4697-9670-de78c1cc0c08" containerID="d2f7174c2222fb60822c99aa9473c07c07d0313fdc6c983adab90c0d8398411f" exitCode=0 Oct 30 06:26:10.943523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:10.943098 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" event={"ID":"22b76b59-27df-4697-9670-de78c1cc0c08","Type":"ContainerDied","Data":"d2f7174c2222fb60822c99aa9473c07c07d0313fdc6c983adab90c0d8398411f"} Oct 30 06:26:12.120188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.120163 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:12.249072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.248958 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-openshift-ca\") pod \"22b76b59-27df-4697-9670-de78c1cc0c08\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " Oct 30 06:26:12.249251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.249087 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-xks-ca\") pod \"22b76b59-27df-4697-9670-de78c1cc0c08\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " Oct 30 06:26:12.249251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.249118 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b76b59-27df-4697-9670-de78c1cc0c08-kube-api-access-nnq75\" (UniqueName: \"kubernetes.io/projected/22b76b59-27df-4697-9670-de78c1cc0c08-kube-api-access-nnq75\") pod \"22b76b59-27df-4697-9670-de78c1cc0c08\" (UID: \"22b76b59-27df-4697-9670-de78c1cc0c08\") " Oct 30 06:26:12.249367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.249340 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-openshift-ca" pod "22b76b59-27df-4697-9670-de78c1cc0c08" (UID: "22b76b59-27df-4697-9670-de78c1cc0c08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:12.250722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.250695 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:12.251637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.251616 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b76b59-27df-4697-9670-de78c1cc0c08-kube-api-access-nnq75" pod "22b76b59-27df-4697-9670-de78c1cc0c08" (UID: "22b76b59-27df-4697-9670-de78c1cc0c08"). InnerVolumeSpecName "kube-api-access-nnq75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:12.251756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.251721 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-xks-ca" pod "22b76b59-27df-4697-9670-de78c1cc0c08" (UID: "22b76b59-27df-4697-9670-de78c1cc0c08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:12.352277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.352238 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b76b59-27df-4697-9670-de78c1cc0c08-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:12.352277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.352271 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnq75\" (UniqueName: \"kubernetes.io/projected/22b76b59-27df-4697-9670-de78c1cc0c08-kube-api-access-nnq75\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:12.966733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.966474 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" Oct 30 06:26:12.966733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.966520 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7" event={"ID":"22b76b59-27df-4697-9670-de78c1cc0c08","Type":"ContainerDied","Data":"2b3f51290ea5f4f911efd3e410555953d4000443f9d21f0e95c84c2adcf4a9e2"} Oct 30 06:26:12.966733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:12.966600 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b3f51290ea5f4f911efd3e410555953d4000443f9d21f0e95c84c2adcf4a9e2" Oct 30 06:26:15.466240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.466204 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22"] Oct 30 06:26:15.466941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.466924 5001 cpu_manager.go:414] "containerMap: removing container" podUID="22b76b59-27df-4697-9670-de78c1cc0c08" containerName="revoke-keys" Oct 30 06:26:15.466984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.466942 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b76b59-27df-4697-9670-de78c1cc0c08" containerName="revoke-keys" Oct 30 06:26:15.467052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.467043 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b76b59-27df-4697-9670-de78c1cc0c08" containerName="revoke-keys" Oct 30 06:26:15.472123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.472103 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:15.486632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.486608 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:26:15.486783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.486659 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:26:15.486783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.486734 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:26:15.486783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.486747 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:26:15.489190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.489164 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22"] Oct 30 06:26:15.595101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.595068 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdqdh\" (UniqueName: \"kubernetes.io/projected/e99cf375-73be-437b-8317-399867eea0ac-kube-api-access-pdqdh\") pod \"maas-api-revoke-sub-e50ddb1c-cwj22\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:15.595281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.595119 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99cf375-73be-437b-8317-399867eea0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e50ddb1c-cwj22\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:15.595281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.595236 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99cf375-73be-437b-8317-399867eea0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e50ddb1c-cwj22\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:15.696748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.696704 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99cf375-73be-437b-8317-399867eea0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e50ddb1c-cwj22\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:15.696954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.696901 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdqdh\" (UniqueName: \"kubernetes.io/projected/e99cf375-73be-437b-8317-399867eea0ac-kube-api-access-pdqdh\") pod \"maas-api-revoke-sub-e50ddb1c-cwj22\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:15.696954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.696933 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99cf375-73be-437b-8317-399867eea0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e50ddb1c-cwj22\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:15.697617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.697591 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99cf375-73be-437b-8317-399867eea0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e50ddb1c-cwj22\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:15.699277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.699252 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99cf375-73be-437b-8317-399867eea0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e50ddb1c-cwj22\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:15.724250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.724187 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdqdh\" (UniqueName: \"kubernetes.io/projected/e99cf375-73be-437b-8317-399867eea0ac-kube-api-access-pdqdh\") pod \"maas-api-revoke-sub-e50ddb1c-cwj22\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:15.787918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:15.787878 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:16.110697 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:16.110670 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99cf375_73be_437b_8317_399867eea0ac.slice/crio-a16ef8b27ceaed3a6a9bea89d86c285fa994cb465927136e3b5f6b73fb1b3810 WatchSource:0}: Error finding container a16ef8b27ceaed3a6a9bea89d86c285fa994cb465927136e3b5f6b73fb1b3810: Status 404 returned error can't find the container with id a16ef8b27ceaed3a6a9bea89d86c285fa994cb465927136e3b5f6b73fb1b3810 Oct 30 06:26:16.120808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:16.120786 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22"] Oct 30 06:26:17.019682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:17.019649 5001 generic.go:356] "Generic (PLEG): container finished" podID="e99cf375-73be-437b-8317-399867eea0ac" containerID="59e99048e70f707cbe16c44b9dd729030ae6014ba5db0e58cdff34bfab19ab7f" exitCode=0 Oct 30 06:26:17.020131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:17.019723 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" event={"ID":"e99cf375-73be-437b-8317-399867eea0ac","Type":"ContainerDied","Data":"59e99048e70f707cbe16c44b9dd729030ae6014ba5db0e58cdff34bfab19ab7f"} Oct 30 06:26:17.020131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:17.019800 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" event={"ID":"e99cf375-73be-437b-8317-399867eea0ac","Type":"ContainerStarted","Data":"a16ef8b27ceaed3a6a9bea89d86c285fa994cb465927136e3b5f6b73fb1b3810"} Oct 30 06:26:18.208076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:18.208051 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:18.340582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:18.340500 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99cf375-73be-437b-8317-399867eea0ac-kube-api-access-pdqdh\" (UniqueName: \"kubernetes.io/projected/e99cf375-73be-437b-8317-399867eea0ac-kube-api-access-pdqdh\") pod \"e99cf375-73be-437b-8317-399867eea0ac\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " Oct 30 06:26:18.340751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:18.340657 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99cf375-73be-437b-8317-399867eea0ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99cf375-73be-437b-8317-399867eea0ac-maas-api-xks-ca\") pod \"e99cf375-73be-437b-8317-399867eea0ac\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " Oct 30 06:26:18.340820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:18.340772 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99cf375-73be-437b-8317-399867eea0ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99cf375-73be-437b-8317-399867eea0ac-maas-api-openshift-ca\") pod \"e99cf375-73be-437b-8317-399867eea0ac\" (UID: \"e99cf375-73be-437b-8317-399867eea0ac\") " Oct 30 06:26:18.341174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:18.341142 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99cf375-73be-437b-8317-399867eea0ac-maas-api-openshift-ca" pod "e99cf375-73be-437b-8317-399867eea0ac" (UID: "e99cf375-73be-437b-8317-399867eea0ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:18.342885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:18.342860 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99cf375-73be-437b-8317-399867eea0ac-maas-api-xks-ca" pod "e99cf375-73be-437b-8317-399867eea0ac" (UID: "e99cf375-73be-437b-8317-399867eea0ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:18.343014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:18.342998 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99cf375-73be-437b-8317-399867eea0ac-kube-api-access-pdqdh" pod "e99cf375-73be-437b-8317-399867eea0ac" (UID: "e99cf375-73be-437b-8317-399867eea0ac"). InnerVolumeSpecName "kube-api-access-pdqdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:18.442653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:18.442622 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99cf375-73be-437b-8317-399867eea0ac-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:18.442653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:18.442650 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99cf375-73be-437b-8317-399867eea0ac-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:18.442653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:18.442660 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdqdh\" (UniqueName: \"kubernetes.io/projected/e99cf375-73be-437b-8317-399867eea0ac-kube-api-access-pdqdh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.039942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:19.039903 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" Oct 30 06:26:19.040126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:19.039946 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22" event={"ID":"e99cf375-73be-437b-8317-399867eea0ac","Type":"ContainerDied","Data":"a16ef8b27ceaed3a6a9bea89d86c285fa994cb465927136e3b5f6b73fb1b3810"} Oct 30 06:26:19.040126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:19.040007 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a16ef8b27ceaed3a6a9bea89d86c285fa994cb465927136e3b5f6b73fb1b3810" Oct 30 06:26:20.947097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:20.947068 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 06:26:20.947558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:20.947146 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 06:26:22.274985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.274947 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z"] Oct 30 06:26:22.275639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.275625 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e99cf375-73be-437b-8317-399867eea0ac" containerName="revoke-keys" Oct 30 06:26:22.275639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.275640 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99cf375-73be-437b-8317-399867eea0ac" containerName="revoke-keys" Oct 30 06:26:22.275760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.275748 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99cf375-73be-437b-8317-399867eea0ac" containerName="revoke-keys" Oct 30 06:26:22.280002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.279979 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.308379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.308348 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z"] Oct 30 06:26:22.449342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.447444 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.449342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.447530 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.449342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.447581 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.449342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.447655 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krft7\" (UniqueName: \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-kube-api-access-krft7\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.449342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.447687 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.449342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.447814 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.449342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.447889 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.449342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.447928 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.449342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.448002 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.549519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.549435 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.549519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.549510 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.549792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.549539 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.549792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.549600 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.549792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.549634 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.549792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.549682 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.549792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.549726 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.550073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.549796 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krft7\" (UniqueName: \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-kube-api-access-krft7\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.550073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.549829 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.550073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.549938 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.550073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.550027 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.550073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.550060 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.550371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.550330 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.550438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.550376 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.552884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.552586 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.552884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.552817 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.576411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.576381 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.579036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.579013 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krft7\" (UniqueName: \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-kube-api-access-krft7\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.595036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.595008 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:22.924975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.924942 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z"] Oct 30 06:26:22.926275 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:22.926248 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be6aa98_42b5_46d3_95bc_a69ee605b47e.slice/crio-5154a0fad2b91c4454135957feec003bdbd206c787cc9f0715f73ebe066753df WatchSource:0}: Error finding container 5154a0fad2b91c4454135957feec003bdbd206c787cc9f0715f73ebe066753df: Status 404 returned error can't find the container with id 5154a0fad2b91c4454135957feec003bdbd206c787cc9f0715f73ebe066753df Oct 30 06:26:22.928782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.928751 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:22.928865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.928818 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:22.928865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:22.928848 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:23.151526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:23.151487 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" event={"ID":"7be6aa98-42b5-46d3-95bc-a69ee605b47e","Type":"ContainerStarted","Data":"3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6"} Oct 30 06:26:23.151743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:23.151548 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" event={"ID":"7be6aa98-42b5-46d3-95bc-a69ee605b47e","Type":"ContainerStarted","Data":"5154a0fad2b91c4454135957feec003bdbd206c787cc9f0715f73ebe066753df"} Oct 30 06:26:23.202496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:23.202389 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" podStartSLOduration=1.202374073 podStartE2EDuration="1.202374073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:23.195525602 +0000 UTC m=+1997.191482612" watchObservedRunningTime="2026-10-30 06:26:23.202374073 +0000 UTC m=+1997.198331083" Oct 30 06:26:23.595890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:23.595794 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:24.598999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:24.598967 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 06:26:24.599462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:24.599034 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 06:26:24.841418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:24.841379 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj"] Oct 30 06:26:24.845953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:24.845937 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:24.880840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:24.880607 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:26:24.880840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:24.880743 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:26:24.880840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:24.880805 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:26:24.896921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:24.896895 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj"] Oct 30 06:26:24.947398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:24.947365 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:26:25.001408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.001365 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml5s5\" (UniqueName: \"kubernetes.io/projected/3bc245d4-e56a-4578-b3d5-ea90cb95388a-kube-api-access-ml5s5\") pod \"maas-api-revoke-sub-257bae11-dxjnj\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:25.001717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.001699 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-257bae11-dxjnj\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:25.002081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.002063 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-257bae11-dxjnj\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:25.102996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.102958 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-257bae11-dxjnj\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:25.103182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.103031 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml5s5\" (UniqueName: \"kubernetes.io/projected/3bc245d4-e56a-4578-b3d5-ea90cb95388a-kube-api-access-ml5s5\") pod \"maas-api-revoke-sub-257bae11-dxjnj\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:25.103182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.103067 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-257bae11-dxjnj\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:25.104047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.104020 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-257bae11-dxjnj\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:25.106197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.106176 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-257bae11-dxjnj\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:25.126365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.126299 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml5s5\" (UniqueName: \"kubernetes.io/projected/3bc245d4-e56a-4578-b3d5-ea90cb95388a-kube-api-access-ml5s5\") pod \"maas-api-revoke-sub-257bae11-dxjnj\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:25.210866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.210598 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:25.400536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.399056 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4"] Oct 30 06:26:25.400536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.399271 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" containerID="cri-o://64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94" gracePeriod=30 Oct 30 06:26:25.602330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.595648 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 06:26:25.602330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.595708 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 06:26:25.697283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:25.696695 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj"] Oct 30 06:26:26.241726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.241691 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" event={"ID":"3bc245d4-e56a-4578-b3d5-ea90cb95388a","Type":"ContainerStarted","Data":"f64ae9622d07642a2f9b1b0f5acf492ab459bba17cac085dcd293314db50efa9"} Oct 30 06:26:26.371329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.370620 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw"] Oct 30 06:26:26.390698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.390672 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:26.397639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.397437 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw"] Oct 30 06:26:26.586449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.578208 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:26.586449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.578283 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:26.586449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.578366 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmrf\" (UniqueName: \"kubernetes.io/projected/e4a69dee-a787-4f09-ab71-0a024beaedc6-kube-api-access-ctmrf\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:26.595529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.595495 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 06:26:26.595798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.595558 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 06:26:26.697383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.696290 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:26.697383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.696387 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:26.697383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.696445 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmrf\" (UniqueName: \"kubernetes.io/projected/e4a69dee-a787-4f09-ab71-0a024beaedc6-kube-api-access-ctmrf\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:26.697994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.697750 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:26.713480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.713445 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:26.721046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.721014 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmrf\" (UniqueName: \"kubernetes.io/projected/e4a69dee-a787-4f09-ab71-0a024beaedc6-kube-api-access-ctmrf\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:26.788587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.788549 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:26.907748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.907712 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd"] Oct 30 06:26:26.916917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.915405 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:26.944008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.943976 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd"] Oct 30 06:26:26.945746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:26.945721 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-645f2b-openshift-default-dockercfg-sgrcj\"" Oct 30 06:26:27.186695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.136746 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-socket\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.186695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.136890 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltxfk\" (UniqueName: \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-kube-api-access-ltxfk\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.186695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.137005 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-envoy\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.186695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.137053 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-certs\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.186695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.137081 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-token\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.186695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.137157 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-data\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.186695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.137210 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-credential-socket\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.186695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.137248 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-podinfo\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.186695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.137352 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2a909-e267-4a73-89a1-37170fa5bc73-istiod-ca-cert\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.331768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.244383 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-podinfo\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.331768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.244504 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2a909-e267-4a73-89a1-37170fa5bc73-istiod-ca-cert\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.331768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.244671 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-socket\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.331768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.244810 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltxfk\" (UniqueName: \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-kube-api-access-ltxfk\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.331768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.244933 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-envoy\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.331768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.244983 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-certs\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.331768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.245010 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-token\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.331768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.245089 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-data\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.331768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.245143 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-credential-socket\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.331768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.245684 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-credential-socket\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.345402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.344686 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-certs\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.348391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.345990 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-data\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.348391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.346484 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-podinfo\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.348391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.347150 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-socket\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.348391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.347548 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2a909-e267-4a73-89a1-37170fa5bc73-istiod-ca-cert\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.358380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.354522 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-envoy\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.409347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.406820 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-token\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.411595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.411532 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltxfk\" (UniqueName: \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-kube-api-access-ltxfk\") pod \"e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.413784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.413372 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" event={"ID":"3bc245d4-e56a-4578-b3d5-ea90cb95388a","Type":"ContainerStarted","Data":"c26cc63e7b03fce5eb3107bf73beb9b656f3a52d174f0afd851ac86d0fb473c4"} Oct 30 06:26:27.532898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.532872 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw"] Oct 30 06:26:27.535039 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:27.535008 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a69dee_a787_4f09_ab71_0a024beaedc6.slice/crio-e557503a4b29d3e43ead58b35637098f1dd1c83cf78916444b32c5f824e5e8d0 WatchSource:0}: Error finding container e557503a4b29d3e43ead58b35637098f1dd1c83cf78916444b32c5f824e5e8d0: Status 404 returned error can't find the container with id e557503a4b29d3e43ead58b35637098f1dd1c83cf78916444b32c5f824e5e8d0 Oct 30 06:26:27.550352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.550032 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:27.596244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.596131 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 06:26:27.596244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.596173 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 06:26:27.992325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:27.992266 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd"] Oct 30 06:26:28.003442 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:28.003409 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f2a909_e267_4a73_89a1_37170fa5bc73.slice/crio-c4641f16543f9924b2b667ed3cef5c03654c946ad832128f836f86bc78047a69 WatchSource:0}: Error finding container c4641f16543f9924b2b667ed3cef5c03654c946ad832128f836f86bc78047a69: Status 404 returned error can't find the container with id c4641f16543f9924b2b667ed3cef5c03654c946ad832128f836f86bc78047a69 Oct 30 06:26:28.016990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.016943 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:28.017232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.017207 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:28.017464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.017425 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:28.431083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.431041 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" event={"ID":"c7f2a909-e267-4a73-89a1-37170fa5bc73","Type":"ContainerStarted","Data":"e88bddbf3606f0ae1de8c3391244724846231af8060b886a17b2e2e54189ec41"} Oct 30 06:26:28.431083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.431107 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" event={"ID":"c7f2a909-e267-4a73-89a1-37170fa5bc73","Type":"ContainerStarted","Data":"c4641f16543f9924b2b667ed3cef5c03654c946ad832128f836f86bc78047a69"} Oct 30 06:26:28.433352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.433156 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" event={"ID":"e4a69dee-a787-4f09-ab71-0a024beaedc6","Type":"ContainerStarted","Data":"5e667b10e6fa190738fad88796fd76daae0ab594a471be5752124129a06b9205"} Oct 30 06:26:28.433352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.433216 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" event={"ID":"e4a69dee-a787-4f09-ab71-0a024beaedc6","Type":"ContainerStarted","Data":"e557503a4b29d3e43ead58b35637098f1dd1c83cf78916444b32c5f824e5e8d0"} Oct 30 06:26:28.438668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.437766 5001 generic.go:356] "Generic (PLEG): container finished" podID="3bc245d4-e56a-4578-b3d5-ea90cb95388a" containerID="c26cc63e7b03fce5eb3107bf73beb9b656f3a52d174f0afd851ac86d0fb473c4" exitCode=0 Oct 30 06:26:28.438668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.437932 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" event={"ID":"3bc245d4-e56a-4578-b3d5-ea90cb95388a","Type":"ContainerDied","Data":"c26cc63e7b03fce5eb3107bf73beb9b656f3a52d174f0afd851ac86d0fb473c4"} Oct 30 06:26:28.563393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.551921 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" podStartSLOduration=2.551887197 podStartE2EDuration="2.551887197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:28.505657724 +0000 UTC m=+2002.501614736" watchObservedRunningTime="2026-10-30 06:26:28.551887197 +0000 UTC m=+2002.547844209" Oct 30 06:26:28.563393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.554397 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:28.596809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.595580 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 06:26:28.596809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.595638 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 06:26:28.609330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:28.597144 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" podStartSLOduration=2.597124592 podStartE2EDuration="2.597124592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:28.595272766 +0000 UTC m=+2002.591229777" watchObservedRunningTime="2026-10-30 06:26:28.597124592 +0000 UTC m=+2002.593081604" Oct 30 06:26:29.287330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.265686 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:29.339971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.339936 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc245d4-e56a-4578-b3d5-ea90cb95388a-kube-api-access-ml5s5\" (UniqueName: \"kubernetes.io/projected/3bc245d4-e56a-4578-b3d5-ea90cb95388a-kube-api-access-ml5s5\") pod \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " Oct 30 06:26:29.340277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.340257 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-openshift-ca\") pod \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " Oct 30 06:26:29.341374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.340751 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-openshift-ca" pod "3bc245d4-e56a-4578-b3d5-ea90cb95388a" (UID: "3bc245d4-e56a-4578-b3d5-ea90cb95388a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:29.343991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.343960 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc245d4-e56a-4578-b3d5-ea90cb95388a-kube-api-access-ml5s5" pod "3bc245d4-e56a-4578-b3d5-ea90cb95388a" (UID: "3bc245d4-e56a-4578-b3d5-ea90cb95388a"). InnerVolumeSpecName "kube-api-access-ml5s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:29.443139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.443103 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-xks-ca\") pod \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\" (UID: \"3bc245d4-e56a-4578-b3d5-ea90cb95388a\") " Oct 30 06:26:29.443804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.443766 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml5s5\" (UniqueName: \"kubernetes.io/projected/3bc245d4-e56a-4578-b3d5-ea90cb95388a-kube-api-access-ml5s5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:29.443804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.443794 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:29.446143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.446114 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-xks-ca" pod "3bc245d4-e56a-4578-b3d5-ea90cb95388a" (UID: "3bc245d4-e56a-4578-b3d5-ea90cb95388a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:29.451168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.451142 5001 generic.go:356] "Generic (PLEG): container finished" podID="e4a69dee-a787-4f09-ab71-0a024beaedc6" containerID="5e667b10e6fa190738fad88796fd76daae0ab594a471be5752124129a06b9205" exitCode=0 Oct 30 06:26:29.451275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.451176 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" event={"ID":"e4a69dee-a787-4f09-ab71-0a024beaedc6","Type":"ContainerDied","Data":"5e667b10e6fa190738fad88796fd76daae0ab594a471be5752124129a06b9205"} Oct 30 06:26:29.502216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.498217 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" Oct 30 06:26:29.502216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.498265 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj" event={"ID":"3bc245d4-e56a-4578-b3d5-ea90cb95388a","Type":"ContainerDied","Data":"f64ae9622d07642a2f9b1b0f5acf492ab459bba17cac085dcd293314db50efa9"} Oct 30 06:26:29.502216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.498349 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f64ae9622d07642a2f9b1b0f5acf492ab459bba17cac085dcd293314db50efa9" Oct 30 06:26:29.547033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.547000 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc245d4-e56a-4578-b3d5-ea90cb95388a-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:29.565338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.561630 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:29.598818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.597403 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 06:26:29.598818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.597457 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 06:26:29.673212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.673110 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms"] Oct 30 06:26:29.674264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.674235 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3bc245d4-e56a-4578-b3d5-ea90cb95388a" containerName="revoke-keys" Oct 30 06:26:29.674264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.674262 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc245d4-e56a-4578-b3d5-ea90cb95388a" containerName="revoke-keys" Oct 30 06:26:29.674509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.674468 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc245d4-e56a-4578-b3d5-ea90cb95388a" containerName="revoke-keys" Oct 30 06:26:29.678661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.678640 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:29.697714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.697681 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms"] Oct 30 06:26:29.891331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.888659 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:29.891331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.888710 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:29.891331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:29.888760 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g52t\" (UniqueName: \"kubernetes.io/projected/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-kube-api-access-5g52t\") pod \"maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:30.016391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.013875 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:30.016391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.013929 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:30.023838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.023773 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:30.024150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.024126 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g52t\" (UniqueName: \"kubernetes.io/projected/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-kube-api-access-5g52t\") pod \"maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:30.026208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.026182 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:30.047557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.047526 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g52t\" (UniqueName: \"kubernetes.io/projected/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-kube-api-access-5g52t\") pod \"maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:30.305395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.304620 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:30.540964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.540649 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:30.541518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.541476 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:26:30.618433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.616518 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 06:26:30.618433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.616578 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 06:26:30.906583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.906491 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms"] Oct 30 06:26:30.970183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:30.967847 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:31.002566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.002480 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:26:31.008177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.008142 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-xks-ca\") pod \"e4a69dee-a787-4f09-ab71-0a024beaedc6\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " Oct 30 06:26:31.008434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.008410 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-openshift-ca\") pod \"e4a69dee-a787-4f09-ab71-0a024beaedc6\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " Oct 30 06:26:31.008692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.008672 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a69dee-a787-4f09-ab71-0a024beaedc6-kube-api-access-ctmrf\" (UniqueName: \"kubernetes.io/projected/e4a69dee-a787-4f09-ab71-0a024beaedc6-kube-api-access-ctmrf\") pod \"e4a69dee-a787-4f09-ab71-0a024beaedc6\" (UID: \"e4a69dee-a787-4f09-ab71-0a024beaedc6\") " Oct 30 06:26:31.009099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.008769 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-openshift-ca" pod "e4a69dee-a787-4f09-ab71-0a024beaedc6" (UID: "e4a69dee-a787-4f09-ab71-0a024beaedc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:31.012359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.009609 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.022925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.021449 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a69dee-a787-4f09-ab71-0a024beaedc6-kube-api-access-ctmrf" pod "e4a69dee-a787-4f09-ab71-0a024beaedc6" (UID: "e4a69dee-a787-4f09-ab71-0a024beaedc6"). InnerVolumeSpecName "kube-api-access-ctmrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:31.022925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.021538 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-xks-ca" pod "e4a69dee-a787-4f09-ab71-0a024beaedc6" (UID: "e4a69dee-a787-4f09-ab71-0a024beaedc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:31.112970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.112939 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c83045-0841-4f12-a77c-ada2ae280906-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c83045-0841-4f12-a77c-ada2ae280906-istiod-ca-cert\") pod \"92c83045-0841-4f12-a77c-ada2ae280906\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " Oct 30 06:26:31.113188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.113001 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92c83045-0841-4f12-a77c-ada2ae280906-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c83045-0841-4f12-a77c-ada2ae280906-istio-podinfo\") pod \"92c83045-0841-4f12-a77c-ada2ae280906\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " Oct 30 06:26:31.113188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.113087 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-envoy\") pod \"92c83045-0841-4f12-a77c-ada2ae280906\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " Oct 30 06:26:31.113188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.113142 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-socket\") pod \"92c83045-0841-4f12-a77c-ada2ae280906\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " Oct 30 06:26:31.113188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.113168 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-istio-token\" (UniqueName: \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-istio-token\") pod \"92c83045-0841-4f12-a77c-ada2ae280906\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " Oct 30 06:26:31.113474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.113199 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-certs\") pod \"92c83045-0841-4f12-a77c-ada2ae280906\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " Oct 30 06:26:31.113474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.113330 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-kube-api-access-snnn2\" (UniqueName: \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-kube-api-access-snnn2\") pod \"92c83045-0841-4f12-a77c-ada2ae280906\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " Oct 30 06:26:31.113474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.113355 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c83045-0841-4f12-a77c-ada2ae280906-istiod-ca-cert" pod "92c83045-0841-4f12-a77c-ada2ae280906" (UID: "92c83045-0841-4f12-a77c-ada2ae280906"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:31.113474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.113365 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-credential-socket\") pod \"92c83045-0841-4f12-a77c-ada2ae280906\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " Oct 30 06:26:31.113474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.113452 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-data\") pod \"92c83045-0841-4f12-a77c-ada2ae280906\" (UID: \"92c83045-0841-4f12-a77c-ada2ae280906\") " Oct 30 06:26:31.113761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.113546 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-credential-socket" pod "92c83045-0841-4f12-a77c-ada2ae280906" (UID: "92c83045-0841-4f12-a77c-ada2ae280906"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:31.114460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.114098 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c83045-0841-4f12-a77c-ada2ae280906-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.114460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.114117 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-certs" pod "92c83045-0841-4f12-a77c-ada2ae280906" (UID: "92c83045-0841-4f12-a77c-ada2ae280906"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:31.114460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.114128 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctmrf\" (UniqueName: \"kubernetes.io/projected/e4a69dee-a787-4f09-ab71-0a024beaedc6-kube-api-access-ctmrf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.114460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.114249 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a69dee-a787-4f09-ab71-0a024beaedc6-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.114460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.114359 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.114460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.114369 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-socket" pod "92c83045-0841-4f12-a77c-ada2ae280906" (UID: "92c83045-0841-4f12-a77c-ada2ae280906"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:31.114460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.114445 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-data" pod "92c83045-0841-4f12-a77c-ada2ae280906" (UID: "92c83045-0841-4f12-a77c-ada2ae280906"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:31.153041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.152968 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92c83045-0841-4f12-a77c-ada2ae280906-istio-podinfo" pod "92c83045-0841-4f12-a77c-ada2ae280906" (UID: "92c83045-0841-4f12-a77c-ada2ae280906"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:26:31.153041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.152978 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-kube-api-access-snnn2" pod "92c83045-0841-4f12-a77c-ada2ae280906" (UID: "92c83045-0841-4f12-a77c-ada2ae280906"). InnerVolumeSpecName "kube-api-access-snnn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:31.153293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.153066 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-envoy" pod "92c83045-0841-4f12-a77c-ada2ae280906" (UID: "92c83045-0841-4f12-a77c-ada2ae280906"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:31.153599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.153567 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-istio-token" pod "92c83045-0841-4f12-a77c-ada2ae280906" (UID: "92c83045-0841-4f12-a77c-ada2ae280906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:31.215886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.215841 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snnn2\" (UniqueName: \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-kube-api-access-snnn2\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.215886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.215880 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.215886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.215897 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c83045-0841-4f12-a77c-ada2ae280906-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.215886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.215918 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.215886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.215931 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.215886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.215943 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c83045-0841-4f12-a77c-ada2ae280906-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.215886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.215955 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c83045-0841-4f12-a77c-ada2ae280906-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:31.548382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.548352 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" Oct 30 06:26:31.548828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.548352 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw" event={"ID":"e4a69dee-a787-4f09-ab71-0a024beaedc6","Type":"ContainerDied","Data":"e557503a4b29d3e43ead58b35637098f1dd1c83cf78916444b32c5f824e5e8d0"} Oct 30 06:26:31.548828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.548484 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e557503a4b29d3e43ead58b35637098f1dd1c83cf78916444b32c5f824e5e8d0" Oct 30 06:26:31.550721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.550696 5001 generic.go:356] "Generic (PLEG): container finished" podID="92c83045-0841-4f12-a77c-ada2ae280906" containerID="64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94" exitCode=0 Oct 30 06:26:31.550911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.550895 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" event={"ID":"92c83045-0841-4f12-a77c-ada2ae280906","Type":"ContainerDied","Data":"64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94"} Oct 30 06:26:31.550953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.550940 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" event={"ID":"92c83045-0841-4f12-a77c-ada2ae280906","Type":"ContainerDied","Data":"2439c30daa77722a92a03306140ebc2ab575dc2eff490814ec66361d53af7db3"} Oct 30 06:26:31.550991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.550972 5001 scope.go:122] "RemoveContainer" containerID="64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94" Oct 30 06:26:31.551162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.551150 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" Oct 30 06:26:31.556931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.556840 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" event={"ID":"4d687a80-28cf-483b-bbf3-3bc6635fbc8c","Type":"ContainerStarted","Data":"5d940b09b950eca0697ad2acc8c26e19e063fa94edbe23bc282149d0ae595d63"} Oct 30 06:26:31.556931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.556890 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" event={"ID":"4d687a80-28cf-483b-bbf3-3bc6635fbc8c","Type":"ContainerStarted","Data":"6bf93ee06f904fb087a248cfa39c61d501d3ae692d9e0357db049370423fb524"} Oct 30 06:26:31.568777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.568695 5001 scope.go:122] "RemoveContainer" containerID="5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460" Oct 30 06:26:31.580768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.580742 5001 scope.go:122] "RemoveContainer" containerID="64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94" Oct 30 06:26:31.581094 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:26:31.581065 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94\": container with ID starting with 64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94 not found: ID does not exist" containerID="64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94" Oct 30 06:26:31.581153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.581107 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94"} err="failed to get container status \"64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94\": rpc error: code = NotFound desc = could not find container \"64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94\": container with ID starting with 64012e3708a66ea8a3326683c7f39483fafcd7a8f367327a21a9597f9e23be94 not found: ID does not exist" Oct 30 06:26:31.581153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.581132 5001 scope.go:122] "RemoveContainer" containerID="5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460" Oct 30 06:26:31.581457 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:26:31.581432 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460\": container with ID starting with 5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460 not found: ID does not exist" containerID="5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460" Oct 30 06:26:31.581547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.581464 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460"} err="failed to get container status \"5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460\": rpc error: code = NotFound desc = could not find container \"5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460\": container with ID starting with 5e16cf9040d99073aa7279d5932767b107751d10bcbf133d75217e1e3818c460 not found: ID does not exist" Oct 30 06:26:31.614110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.614040 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" podStartSLOduration=2.614021653 podStartE2EDuration="2.614021653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:31.610300108 +0000 UTC m=+2005.606257118" watchObservedRunningTime="2026-10-30 06:26:31.614021653 +0000 UTC m=+2005.609978666" Oct 30 06:26:31.670773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.670735 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4"] Oct 30 06:26:31.679453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.679423 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4"] Oct 30 06:26:31.957492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.957436 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:31.957686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:31.957526 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-dfa2ae-openshift-default-cfbb6844f-q8fj4" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:32.122834 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.122792 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c83045-0841-4f12-a77c-ada2ae280906" path="/var/lib/kubelet/pods/92c83045-0841-4f12-a77c-ada2ae280906/volumes" Oct 30 06:26:32.583330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.582628 5001 generic.go:356] "Generic (PLEG): container finished" podID="4d687a80-28cf-483b-bbf3-3bc6635fbc8c" containerID="5d940b09b950eca0697ad2acc8c26e19e063fa94edbe23bc282149d0ae595d63" exitCode=0 Oct 30 06:26:32.583330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.582775 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" event={"ID":"4d687a80-28cf-483b-bbf3-3bc6635fbc8c","Type":"ContainerDied","Data":"5d940b09b950eca0697ad2acc8c26e19e063fa94edbe23bc282149d0ae595d63"} Oct 30 06:26:32.637476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.637434 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:32.637665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.637482 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:32.828065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.828033 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:32.828578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.828565 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:32.829434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.829411 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:26:32.906031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.905994 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5"] Oct 30 06:26:32.906787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.906765 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e4a69dee-a787-4f09-ab71-0a024beaedc6" containerName="revoke-keys" Oct 30 06:26:32.906787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.906784 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a69dee-a787-4f09-ab71-0a024beaedc6" containerName="revoke-keys" Oct 30 06:26:32.906990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.906799 5001 cpu_manager.go:414] "containerMap: removing container" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" Oct 30 06:26:32.906990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.906805 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" Oct 30 06:26:32.906990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.906818 5001 cpu_manager.go:414] "containerMap: removing container" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" Oct 30 06:26:32.906990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.906826 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" Oct 30 06:26:32.906990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.906943 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a69dee-a787-4f09-ab71-0a024beaedc6" containerName="revoke-keys" Oct 30 06:26:32.906990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.906952 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" Oct 30 06:26:32.906990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.906957 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c83045-0841-4f12-a77c-ada2ae280906" containerName="istio-proxy" Oct 30 06:26:32.910852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.910829 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 06:26:32.943491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.943464 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-645f2b-dockercfg-zqvwv\"" Oct 30 06:26:32.944034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:32.944009 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-645f2b\"" Oct 30 06:26:33.052350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.043529 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pvlq\" (UniqueName: \"kubernetes.io/projected/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-kube-api-access-5pvlq\") pod \"payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5\" (UID: \"5d03a7c2-0012-42dd-afd3-03c0d3c468e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 06:26:33.052350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.043770 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5\" (UID: \"5d03a7c2-0012-42dd-afd3-03c0d3c468e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 06:26:33.191385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.170531 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pvlq\" (UniqueName: \"kubernetes.io/projected/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-kube-api-access-5pvlq\") pod \"payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5\" (UID: \"5d03a7c2-0012-42dd-afd3-03c0d3c468e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 06:26:33.191385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.170705 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5\" (UID: \"5d03a7c2-0012-42dd-afd3-03c0d3c468e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 06:26:33.234344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.231632 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5\" (UID: \"5d03a7c2-0012-42dd-afd3-03c0d3c468e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 06:26:33.261322 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.256743 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5"] Oct 30 06:26:33.297561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.289804 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2"] Oct 30 06:26:33.347047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.346400 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pvlq\" (UniqueName: \"kubernetes.io/projected/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-kube-api-access-5pvlq\") pod \"payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5\" (UID: \"5d03a7c2-0012-42dd-afd3-03c0d3c468e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 06:26:33.350628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.350597 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 06:26:33.354722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.354691 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2"] Oct 30 06:26:33.446979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.446882 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdfpg\" (UniqueName: \"kubernetes.io/projected/1131c14c-64e3-4840-88b7-50b0bde68434-kube-api-access-sdfpg\") pod \"payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2\" (UID: \"1131c14c-64e3-4840-88b7-50b0bde68434\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 06:26:33.446979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.446954 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1131c14c-64e3-4840-88b7-50b0bde68434-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2\" (UID: \"1131c14c-64e3-4840-88b7-50b0bde68434\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 06:26:33.526215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.526181 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 06:26:33.549126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.547965 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdfpg\" (UniqueName: \"kubernetes.io/projected/1131c14c-64e3-4840-88b7-50b0bde68434-kube-api-access-sdfpg\") pod \"payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2\" (UID: \"1131c14c-64e3-4840-88b7-50b0bde68434\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 06:26:33.549126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.548062 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1131c14c-64e3-4840-88b7-50b0bde68434-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2\" (UID: \"1131c14c-64e3-4840-88b7-50b0bde68434\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 06:26:33.549126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.549073 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1131c14c-64e3-4840-88b7-50b0bde68434-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2\" (UID: \"1131c14c-64e3-4840-88b7-50b0bde68434\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 06:26:33.592017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.591984 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdfpg\" (UniqueName: \"kubernetes.io/projected/1131c14c-64e3-4840-88b7-50b0bde68434-kube-api-access-sdfpg\") pod \"payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2\" (UID: \"1131c14c-64e3-4840-88b7-50b0bde68434\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 06:26:33.686903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.686872 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 06:26:33.877583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.877414 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:33.906166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.906139 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5"] Oct 30 06:26:33.973026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.972936 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-openshift-ca\") pod \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " Oct 30 06:26:33.974236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.974210 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-kube-api-access-5g52t\" (UniqueName: \"kubernetes.io/projected/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-kube-api-access-5g52t\") pod \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " Oct 30 06:26:33.974428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.974401 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-openshift-ca" pod "4d687a80-28cf-483b-bbf3-3bc6635fbc8c" (UID: "4d687a80-28cf-483b-bbf3-3bc6635fbc8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:33.974592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.974571 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-xks-ca\") pod \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\" (UID: \"4d687a80-28cf-483b-bbf3-3bc6635fbc8c\") " Oct 30 06:26:33.975376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.975352 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:33.978057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.978030 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-kube-api-access-5g52t" pod "4d687a80-28cf-483b-bbf3-3bc6635fbc8c" (UID: "4d687a80-28cf-483b-bbf3-3bc6635fbc8c"). InnerVolumeSpecName "kube-api-access-5g52t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:33.981145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:33.981119 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-xks-ca" pod "4d687a80-28cf-483b-bbf3-3bc6635fbc8c" (UID: "4d687a80-28cf-483b-bbf3-3bc6635fbc8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:34.049955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.049922 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2"] Oct 30 06:26:34.052577 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:34.052549 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1131c14c_64e3_4840_88b7_50b0bde68434.slice/crio-bed771c4dd508d862ec5d98d29c311cba479f502d76b7bd22dd7eec1dc46d9b4 WatchSource:0}: Error finding container bed771c4dd508d862ec5d98d29c311cba479f502d76b7bd22dd7eec1dc46d9b4: Status 404 returned error can't find the container with id bed771c4dd508d862ec5d98d29c311cba479f502d76b7bd22dd7eec1dc46d9b4 Oct 30 06:26:34.076901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.076797 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g52t\" (UniqueName: \"kubernetes.io/projected/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-kube-api-access-5g52t\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:34.076901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.076831 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d687a80-28cf-483b-bbf3-3bc6635fbc8c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:34.610715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.610640 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" Oct 30 06:26:34.611160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.610641 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms" event={"ID":"4d687a80-28cf-483b-bbf3-3bc6635fbc8c","Type":"ContainerDied","Data":"6bf93ee06f904fb087a248cfa39c61d501d3ae692d9e0357db049370423fb524"} Oct 30 06:26:34.611160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.610766 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bf93ee06f904fb087a248cfa39c61d501d3ae692d9e0357db049370423fb524" Oct 30 06:26:34.612581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.612555 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" event={"ID":"5d03a7c2-0012-42dd-afd3-03c0d3c468e3","Type":"ContainerStarted","Data":"cb0dc7699158a55cbeb4dc11003206fa8bbdc59eae46e3c362d7f7d5d52caee1"} Oct 30 06:26:34.612703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.612615 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" event={"ID":"5d03a7c2-0012-42dd-afd3-03c0d3c468e3","Type":"ContainerStarted","Data":"6fffe45d4e3029e8f2179cb6144962b5f8b1d0501e2ca4efbfad102756e07c50"} Oct 30 06:26:34.612703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.612649 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 06:26:34.614265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.614242 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" event={"ID":"1131c14c-64e3-4840-88b7-50b0bde68434","Type":"ContainerStarted","Data":"e4cf4464467e1b947aad230ab5e3cfa63fed9bfd34b6bc80eaf02b9c062ab25d"} Oct 30 06:26:34.614422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.614281 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" event={"ID":"1131c14c-64e3-4840-88b7-50b0bde68434","Type":"ContainerStarted","Data":"bed771c4dd508d862ec5d98d29c311cba479f502d76b7bd22dd7eec1dc46d9b4"} Oct 30 06:26:34.614422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.614318 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 06:26:34.664270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.664208 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" podStartSLOduration=2.664190611 podStartE2EDuration="2.664190611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:34.652146507 +0000 UTC m=+2008.648103537" watchObservedRunningTime="2026-10-30 06:26:34.664190611 +0000 UTC m=+2008.660147621" Oct 30 06:26:34.706343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:34.706253 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" podStartSLOduration=1.7062370279999999 podStartE2EDuration="1.706237028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:34.704723915 +0000 UTC m=+2008.700680938" watchObservedRunningTime="2026-10-30 06:26:34.706237028 +0000 UTC m=+2008.702194040" Oct 30 06:26:36.237891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.237858 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n"] Oct 30 06:26:36.238861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.238840 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4d687a80-28cf-483b-bbf3-3bc6635fbc8c" containerName="revoke-keys" Oct 30 06:26:36.238976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.238862 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d687a80-28cf-483b-bbf3-3bc6635fbc8c" containerName="revoke-keys" Oct 30 06:26:36.239081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.239065 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d687a80-28cf-483b-bbf3-3bc6635fbc8c" containerName="revoke-keys" Oct 30 06:26:36.243216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.243196 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.275753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.275710 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-6e2548-openshift-default-dockercfg-rmk8g\"" Oct 30 06:26:36.277272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.277244 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n"] Oct 30 06:26:36.306085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.306037 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-data\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.306290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.306172 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istiod-ca-cert\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.306290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.306224 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-socket\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.306290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.306277 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-certs\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.306523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.306393 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-podinfo\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.306523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.306493 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29l8z\" (UniqueName: \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-kube-api-access-29l8z\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.306650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.306548 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-envoy\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.306650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.306601 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-token\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.306763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.306655 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-credential-socket\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.413336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.412202 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29l8z\" (UniqueName: \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-kube-api-access-29l8z\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.413336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.412271 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-envoy\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.413336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.412334 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-token\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.413336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.412388 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-credential-socket\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.413336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.412468 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-data\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.413336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.412535 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istiod-ca-cert\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.413336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.412565 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-socket\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.413336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.412616 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-certs\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.413336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.412714 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-podinfo\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.433378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.418326 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-credential-socket\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.449522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.419208 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-podinfo\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.449522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.419535 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-data\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.449522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.420142 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istiod-ca-cert\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.449522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.420556 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-socket\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.449522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.420954 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-certs\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.476690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.476651 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-envoy\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.511013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.510903 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-token\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.511928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.511863 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29l8z\" (UniqueName: \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-kube-api-access-29l8z\") pod \"e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.569328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.566470 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:36.910595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.907932 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4"] Oct 30 06:26:36.951797 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:36.951765 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5"] Oct 30 06:26:37.001021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:37.000525 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n"] Oct 30 06:26:37.006501 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:37.006456 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbe1898_1454_48fe_8e9f_c1bcf0600b44.slice/crio-df0f10736befcf3248e906c6beff8f42cc231df78eaadc4917937328f93c3c5d WatchSource:0}: Error finding container df0f10736befcf3248e906c6beff8f42cc231df78eaadc4917937328f93c3c5d: Status 404 returned error can't find the container with id df0f10736befcf3248e906c6beff8f42cc231df78eaadc4917937328f93c3c5d Oct 30 06:26:37.008859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:37.008808 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:37.008989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:37.008908 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:37.008989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:37.008980 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:37.690741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:37.690696 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" event={"ID":"1dbe1898-1454-48fe-8e9f-c1bcf0600b44","Type":"ContainerStarted","Data":"1be51816e7cd52820608b7fc1dc43ca669c6c749a5aedb89cf3d25dcdaf20018"} Oct 30 06:26:37.691333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:37.690776 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" event={"ID":"1dbe1898-1454-48fe-8e9f-c1bcf0600b44","Type":"ContainerStarted","Data":"df0f10736befcf3248e906c6beff8f42cc231df78eaadc4917937328f93c3c5d"} Oct 30 06:26:37.750700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:37.750628 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" podStartSLOduration=1.750613123 podStartE2EDuration="1.750613123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:37.747812054 +0000 UTC m=+2011.743769099" watchObservedRunningTime="2026-10-30 06:26:37.750613123 +0000 UTC m=+2011.746570134" Oct 30 06:26:38.267281 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:26:38.267241 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:26:38.567667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:38.567581 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:38.574921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:38.574896 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:38.700591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:38.700558 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:38.701486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:38.701465 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:26:40.153150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:40.153111 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k"] Oct 30 06:26:40.191661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:40.191624 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2"] Oct 30 06:26:41.908452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:41.908407 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" podUID="1f72ba28-bde1-41eb-88de-b2bdf0ff6fde" containerName="payload-processing" containerID="cri-o://aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677" gracePeriod=25 Oct 30 06:26:41.952253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:41.952198 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" podUID="87082fc2-7177-4533-bb11-3352c0a6bc66" containerName="payload-pre-processing" containerID="cri-o://d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265" gracePeriod=25 Oct 30 06:26:42.258540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.258513 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:26:42.289930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.289905 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:26:42.359770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.359726 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87082fc2-7177-4533-bb11-3352c0a6bc66-kube-api-access-zb4pb\" (UniqueName: \"kubernetes.io/projected/87082fc2-7177-4533-bb11-3352c0a6bc66-kube-api-access-zb4pb\") pod \"87082fc2-7177-4533-bb11-3352c0a6bc66\" (UID: \"87082fc2-7177-4533-bb11-3352c0a6bc66\") " Oct 30 06:26:42.359992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.359957 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87082fc2-7177-4533-bb11-3352c0a6bc66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87082fc2-7177-4533-bb11-3352c0a6bc66-plugins-config-volume\") pod \"87082fc2-7177-4533-bb11-3352c0a6bc66\" (UID: \"87082fc2-7177-4533-bb11-3352c0a6bc66\") " Oct 30 06:26:42.360081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.360051 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-kube-api-access-wp98z\" (UniqueName: \"kubernetes.io/projected/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-kube-api-access-wp98z\") pod \"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde\" (UID: \"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde\") " Oct 30 06:26:42.360169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.360152 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-plugins-config-volume\") pod \"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde\" (UID: \"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde\") " Oct 30 06:26:42.360757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.360722 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87082fc2-7177-4533-bb11-3352c0a6bc66-plugins-config-volume" pod "87082fc2-7177-4533-bb11-3352c0a6bc66" (UID: "87082fc2-7177-4533-bb11-3352c0a6bc66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:42.361086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.361053 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-plugins-config-volume" pod "1f72ba28-bde1-41eb-88de-b2bdf0ff6fde" (UID: "1f72ba28-bde1-41eb-88de-b2bdf0ff6fde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:42.363256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.363234 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-kube-api-access-wp98z" pod "1f72ba28-bde1-41eb-88de-b2bdf0ff6fde" (UID: "1f72ba28-bde1-41eb-88de-b2bdf0ff6fde"). InnerVolumeSpecName "kube-api-access-wp98z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:42.365007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.364984 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87082fc2-7177-4533-bb11-3352c0a6bc66-kube-api-access-zb4pb" pod "87082fc2-7177-4533-bb11-3352c0a6bc66" (UID: "87082fc2-7177-4533-bb11-3352c0a6bc66"). InnerVolumeSpecName "kube-api-access-zb4pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:42.462411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.462330 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:42.462411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.462362 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb4pb\" (UniqueName: \"kubernetes.io/projected/87082fc2-7177-4533-bb11-3352c0a6bc66-kube-api-access-zb4pb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:42.462411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.462373 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87082fc2-7177-4533-bb11-3352c0a6bc66-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:42.462411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.462384 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp98z\" (UniqueName: \"kubernetes.io/projected/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde-kube-api-access-wp98z\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:42.540607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.540572 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m"] Oct 30 06:26:42.541396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.541374 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1f72ba28-bde1-41eb-88de-b2bdf0ff6fde" containerName="payload-processing" Oct 30 06:26:42.541396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.541395 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f72ba28-bde1-41eb-88de-b2bdf0ff6fde" containerName="payload-processing" Oct 30 06:26:42.541604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.541418 5001 cpu_manager.go:414] "containerMap: removing container" podUID="87082fc2-7177-4533-bb11-3352c0a6bc66" containerName="payload-pre-processing" Oct 30 06:26:42.541604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.541427 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87082fc2-7177-4533-bb11-3352c0a6bc66" containerName="payload-pre-processing" Oct 30 06:26:42.541720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.541617 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f72ba28-bde1-41eb-88de-b2bdf0ff6fde" containerName="payload-processing" Oct 30 06:26:42.541720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.541636 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="87082fc2-7177-4533-bb11-3352c0a6bc66" containerName="payload-pre-processing" Oct 30 06:26:42.545733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.545710 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:26:42.563547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.563522 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfpgk\" (UniqueName: \"kubernetes.io/projected/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-kube-api-access-pfpgk\") pod \"payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m\" (UID: \"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:26:42.563691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.563577 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m\" (UID: \"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:26:42.573786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.573762 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-6e2548-dockercfg-xt2bl\"" Oct 30 06:26:42.576205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.576167 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-6e2548\"" Oct 30 06:26:42.584484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.584456 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m"] Oct 30 06:26:42.701350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.698827 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m\" (UID: \"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:26:42.701350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.699231 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpgk\" (UniqueName: \"kubernetes.io/projected/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-kube-api-access-pfpgk\") pod \"payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m\" (UID: \"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:26:42.701350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.700528 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m\" (UID: \"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:26:42.833378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.825929 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfpgk\" (UniqueName: \"kubernetes.io/projected/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-kube-api-access-pfpgk\") pod \"payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m\" (UID: \"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:26:42.936615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:42.936585 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:26:43.091421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.091385 5001 generic.go:356] "Generic (PLEG): container finished" podID="87082fc2-7177-4533-bb11-3352c0a6bc66" containerID="d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265" exitCode=0 Oct 30 06:26:43.101389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.091724 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" event={"ID":"87082fc2-7177-4533-bb11-3352c0a6bc66","Type":"ContainerDied","Data":"d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265"} Oct 30 06:26:43.101900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.101864 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" event={"ID":"87082fc2-7177-4533-bb11-3352c0a6bc66","Type":"ContainerDied","Data":"4c46f0709348a3445fc6273115fd5f4a4bd87e319d726b2fb8d24edb517e5a3a"} Oct 30 06:26:43.102123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.102099 5001 scope.go:122] "RemoveContainer" containerID="d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265" Oct 30 06:26:43.102524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.102506 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5" Oct 30 06:26:43.158414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.158384 5001 generic.go:356] "Generic (PLEG): container finished" podID="1f72ba28-bde1-41eb-88de-b2bdf0ff6fde" containerID="aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677" exitCode=0 Oct 30 06:26:43.159394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.159378 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" Oct 30 06:26:43.167204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.165948 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" event={"ID":"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde","Type":"ContainerDied","Data":"aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677"} Oct 30 06:26:43.167204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.166009 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4" event={"ID":"1f72ba28-bde1-41eb-88de-b2bdf0ff6fde","Type":"ContainerDied","Data":"4012c87cbc3da332237bf4234cf458f689f50ea6a0a5b6add4dfe40248926a44"} Oct 30 06:26:43.245388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.245352 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5"] Oct 30 06:26:43.313359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.306849 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5a6371-86957fc887-7h9c5"] Oct 30 06:26:43.318105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.318068 5001 scope.go:122] "RemoveContainer" containerID="d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265" Oct 30 06:26:43.318873 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:26:43.318847 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265\": container with ID starting with d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265 not found: ID does not exist" containerID="d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265" Oct 30 06:26:43.319084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.319055 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265"} err="failed to get container status \"d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265\": rpc error: code = NotFound desc = could not find container \"d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265\": container with ID starting with d7a9e453f090fbfb25c318e1ce273d5b6f52ce4d97be5f5413dabf9d72ed9265 not found: ID does not exist" Oct 30 06:26:43.319230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.319217 5001 scope.go:122] "RemoveContainer" containerID="aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677" Oct 30 06:26:43.345838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.343637 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4"] Oct 30 06:26:43.353216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.352987 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a6371-84f5455bfb-fkrt4"] Oct 30 06:26:43.357194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.357172 5001 scope.go:122] "RemoveContainer" containerID="aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677" Oct 30 06:26:43.357639 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:26:43.357606 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677\": container with ID starting with aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677 not found: ID does not exist" containerID="aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677" Oct 30 06:26:43.357825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.357795 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677"} err="failed to get container status \"aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677\": rpc error: code = NotFound desc = could not find container \"aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677\": container with ID starting with aba25a6658d3a22a6cd1d8bcb66657c39f28045b2ad584415e326db560ef7677 not found: ID does not exist" Oct 30 06:26:43.608646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.608620 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw"] Oct 30 06:26:43.619386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.619353 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a6371-79ddw"] Oct 30 06:26:43.711592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.711552 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m"] Oct 30 06:26:43.715553 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:43.715518 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5eddac_dbe2_485e_8c7b_1fec89d02eeb.slice/crio-7d45fc6a46af55efb72b51973c4b2e79fed25de4824df0eadf4aed84397df343 WatchSource:0}: Error finding container 7d45fc6a46af55efb72b51973c4b2e79fed25de4824df0eadf4aed84397df343: Status 404 returned error can't find the container with id 7d45fc6a46af55efb72b51973c4b2e79fed25de4824df0eadf4aed84397df343 Oct 30 06:26:43.840330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.836941 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77"] Oct 30 06:26:43.847354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:43.842540 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" containerID="cri-o://36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32" gracePeriod=30 Oct 30 06:26:44.120178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:44.120142 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f72ba28-bde1-41eb-88de-b2bdf0ff6fde" path="/var/lib/kubelet/pods/1f72ba28-bde1-41eb-88de-b2bdf0ff6fde/volumes" Oct 30 06:26:44.120724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:44.120706 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87082fc2-7177-4533-bb11-3352c0a6bc66" path="/var/lib/kubelet/pods/87082fc2-7177-4533-bb11-3352c0a6bc66/volumes" Oct 30 06:26:44.121160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:44.121144 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a69dee-a787-4f09-ab71-0a024beaedc6" path="/var/lib/kubelet/pods/e4a69dee-a787-4f09-ab71-0a024beaedc6/volumes" Oct 30 06:26:44.171622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:44.171582 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" event={"ID":"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb","Type":"ContainerStarted","Data":"596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974"} Oct 30 06:26:44.171814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:44.171648 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" event={"ID":"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb","Type":"ContainerStarted","Data":"7d45fc6a46af55efb72b51973c4b2e79fed25de4824df0eadf4aed84397df343"} Oct 30 06:26:44.171814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:44.171679 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:26:44.220904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:44.220847 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" podStartSLOduration=2.220832346 podStartE2EDuration="2.220832346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:44.215213596 +0000 UTC m=+2018.211170619" watchObservedRunningTime="2026-10-30 06:26:44.220832346 +0000 UTC m=+2018.216789355" Oct 30 06:26:45.154263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.154220 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" podUID="d869917a-e4ec-4340-a5d9-cac52710f4ec" containerName="payload-processing" containerID="cri-o://bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7" gracePeriod=25 Oct 30 06:26:45.192375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.192343 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" podUID="91d94784-4586-492e-9f9c-fcea45efb051" containerName="payload-pre-processing" containerID="cri-o://f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82" gracePeriod=25 Oct 30 06:26:45.635975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.635942 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 06:26:45.637262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.637206 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 06:26:45.792561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.792523 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:26:45.856946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.856919 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:26:45.980569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.980483 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d869917a-e4ec-4340-a5d9-cac52710f4ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d869917a-e4ec-4340-a5d9-cac52710f4ec-plugins-config-volume\") pod \"d869917a-e4ec-4340-a5d9-cac52710f4ec\" (UID: \"d869917a-e4ec-4340-a5d9-cac52710f4ec\") " Oct 30 06:26:45.980569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.980552 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d94784-4586-492e-9f9c-fcea45efb051-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d94784-4586-492e-9f9c-fcea45efb051-plugins-config-volume\") pod \"91d94784-4586-492e-9f9c-fcea45efb051\" (UID: \"91d94784-4586-492e-9f9c-fcea45efb051\") " Oct 30 06:26:45.980782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.980612 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d869917a-e4ec-4340-a5d9-cac52710f4ec-kube-api-access-7rw6t\" (UniqueName: \"kubernetes.io/projected/d869917a-e4ec-4340-a5d9-cac52710f4ec-kube-api-access-7rw6t\") pod \"d869917a-e4ec-4340-a5d9-cac52710f4ec\" (UID: \"d869917a-e4ec-4340-a5d9-cac52710f4ec\") " Oct 30 06:26:45.980782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.980665 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d94784-4586-492e-9f9c-fcea45efb051-kube-api-access-p6r7r\" (UniqueName: \"kubernetes.io/projected/91d94784-4586-492e-9f9c-fcea45efb051-kube-api-access-p6r7r\") pod \"91d94784-4586-492e-9f9c-fcea45efb051\" (UID: \"91d94784-4586-492e-9f9c-fcea45efb051\") " Oct 30 06:26:45.980925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.980896 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d94784-4586-492e-9f9c-fcea45efb051-plugins-config-volume" pod "91d94784-4586-492e-9f9c-fcea45efb051" (UID: "91d94784-4586-492e-9f9c-fcea45efb051"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:45.980925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.980896 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d869917a-e4ec-4340-a5d9-cac52710f4ec-plugins-config-volume" pod "d869917a-e4ec-4340-a5d9-cac52710f4ec" (UID: "d869917a-e4ec-4340-a5d9-cac52710f4ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:45.981224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.981205 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d869917a-e4ec-4340-a5d9-cac52710f4ec-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:45.981281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.981230 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d94784-4586-492e-9f9c-fcea45efb051-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:45.983006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.982982 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d94784-4586-492e-9f9c-fcea45efb051-kube-api-access-p6r7r" pod "91d94784-4586-492e-9f9c-fcea45efb051" (UID: "91d94784-4586-492e-9f9c-fcea45efb051"). InnerVolumeSpecName "kube-api-access-p6r7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:45.983082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:45.983030 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d869917a-e4ec-4340-a5d9-cac52710f4ec-kube-api-access-7rw6t" pod "d869917a-e4ec-4340-a5d9-cac52710f4ec" (UID: "d869917a-e4ec-4340-a5d9-cac52710f4ec"). InnerVolumeSpecName "kube-api-access-7rw6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:46.081989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.081954 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rw6t\" (UniqueName: \"kubernetes.io/projected/d869917a-e4ec-4340-a5d9-cac52710f4ec-kube-api-access-7rw6t\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:46.081989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.081983 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6r7r\" (UniqueName: \"kubernetes.io/projected/91d94784-4586-492e-9f9c-fcea45efb051-kube-api-access-p6r7r\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:46.195608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.195577 5001 generic.go:356] "Generic (PLEG): container finished" podID="d869917a-e4ec-4340-a5d9-cac52710f4ec" containerID="bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7" exitCode=0 Oct 30 06:26:46.196048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.195640 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" Oct 30 06:26:46.196048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.195676 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" event={"ID":"d869917a-e4ec-4340-a5d9-cac52710f4ec","Type":"ContainerDied","Data":"bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7"} Oct 30 06:26:46.196048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.195750 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k" event={"ID":"d869917a-e4ec-4340-a5d9-cac52710f4ec","Type":"ContainerDied","Data":"adbc2dc8f3d8907c6dff0d3d6aded5a53387a55cdc46b82e403af79a1b67fe05"} Oct 30 06:26:46.196048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.195784 5001 scope.go:122] "RemoveContainer" containerID="bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7" Oct 30 06:26:46.197815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.197666 5001 generic.go:356] "Generic (PLEG): container finished" podID="91d94784-4586-492e-9f9c-fcea45efb051" containerID="f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82" exitCode=0 Oct 30 06:26:46.197815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.197715 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" Oct 30 06:26:46.197815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.197731 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" event={"ID":"91d94784-4586-492e-9f9c-fcea45efb051","Type":"ContainerDied","Data":"f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82"} Oct 30 06:26:46.197815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.197773 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2" event={"ID":"91d94784-4586-492e-9f9c-fcea45efb051","Type":"ContainerDied","Data":"a19006327ba763278eac6294b5dd010e1065432b649fdfbf7f87643435f40e46"} Oct 30 06:26:46.208450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.208434 5001 scope.go:122] "RemoveContainer" containerID="bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7" Oct 30 06:26:46.208715 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:26:46.208696 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7\": container with ID starting with bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7 not found: ID does not exist" containerID="bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7" Oct 30 06:26:46.208770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.208723 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7"} err="failed to get container status \"bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7\": rpc error: code = NotFound desc = could not find container \"bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7\": container with ID starting with bdbd9753ea2a8ad0aace6001efbd40d888882b0522bb2cb809a2510a175099e7 not found: ID does not exist" Oct 30 06:26:46.208770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.208741 5001 scope.go:122] "RemoveContainer" containerID="f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82" Oct 30 06:26:46.219782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.219762 5001 scope.go:122] "RemoveContainer" containerID="f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82" Oct 30 06:26:46.220083 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:26:46.220065 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82\": container with ID starting with f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82 not found: ID does not exist" containerID="f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82" Oct 30 06:26:46.220122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.220093 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82"} err="failed to get container status \"f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82\": rpc error: code = NotFound desc = could not find container \"f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82\": container with ID starting with f229b36f3dec57c67323c9d93d56a042e3f7dea0cdcad0a35d53395470284d82 not found: ID does not exist" Oct 30 06:26:46.248923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.248863 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k"] Oct 30 06:26:46.260553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.260526 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f5e55f-7c9f44cdf6-dmn9k"] Oct 30 06:26:46.286295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.286257 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2"] Oct 30 06:26:46.291444 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:46.291419 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f5e55f-d4dd84595-zhrs2"] Oct 30 06:26:47.981581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:47.981483 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95"] Oct 30 06:26:47.982555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:47.982529 5001 cpu_manager.go:414] "containerMap: removing container" podUID="91d94784-4586-492e-9f9c-fcea45efb051" containerName="payload-pre-processing" Oct 30 06:26:47.982732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:47.982708 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d94784-4586-492e-9f9c-fcea45efb051" containerName="payload-pre-processing" Oct 30 06:26:47.982877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:47.982774 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d869917a-e4ec-4340-a5d9-cac52710f4ec" containerName="payload-processing" Oct 30 06:26:47.982877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:47.982782 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d869917a-e4ec-4340-a5d9-cac52710f4ec" containerName="payload-processing" Oct 30 06:26:47.983005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:47.982953 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d869917a-e4ec-4340-a5d9-cac52710f4ec" containerName="payload-processing" Oct 30 06:26:47.983005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:47.982975 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d94784-4586-492e-9f9c-fcea45efb051" containerName="payload-pre-processing" Oct 30 06:26:47.993874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:47.993852 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.013063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.013032 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95"] Oct 30 06:26:48.015206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.015171 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qdmm\" (UniqueName: \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-kube-api-access-4qdmm\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.015405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.015226 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-credential-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.015581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.015558 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-podinfo\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.015729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.015709 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-data\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.015864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.015841 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.015985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.015900 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-token\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.016050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.016005 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istiod-ca-cert\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.016103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.016053 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-certs\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.016103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.016081 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-envoy\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.017681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.017656 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-fa5a69-openshift-default-dockercfg-6rbm2\"" Oct 30 06:26:48.163338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.163181 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.179384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.163616 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-token\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.179384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.165076 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istiod-ca-cert\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.179384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.165131 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istiod-ca-cert\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.179384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.165678 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-certs\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.179384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.174710 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-envoy\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.179384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.174863 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.179384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.175367 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdmm\" (UniqueName: \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-kube-api-access-4qdmm\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.179384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.175484 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-certs\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.179384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.175743 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-credential-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.179384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.176554 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-podinfo\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.179384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.177470 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-credential-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.215176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.177881 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-data\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.215176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.178233 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-data\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.253436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.217794 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-podinfo\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.274345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.260909 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-envoy\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.291331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.288026 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qdmm\" (UniqueName: \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-kube-api-access-4qdmm\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.297668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.295022 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d94784-4586-492e-9f9c-fcea45efb051" path="/var/lib/kubelet/pods/91d94784-4586-492e-9f9c-fcea45efb051/volumes" Oct 30 06:26:48.297668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.295885 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d869917a-e4ec-4340-a5d9-cac52710f4ec" path="/var/lib/kubelet/pods/d869917a-e4ec-4340-a5d9-cac52710f4ec/volumes" Oct 30 06:26:48.347655 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.344066 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-token\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.402953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.396690 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:48.473386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.473357 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms"] Oct 30 06:26:48.480746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.480717 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f5e55f-sppms"] Oct 30 06:26:48.896225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.896196 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95"] Oct 30 06:26:48.958551 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:48.958519 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc22a5c_aff0_46dc_be59_b438bffb07a9.slice/crio-0b4e0eb7a9642f0ffe1560b9fd60b5b8f8c78e1f312248318afd29f8a4be3a01 WatchSource:0}: Error finding container 0b4e0eb7a9642f0ffe1560b9fd60b5b8f8c78e1f312248318afd29f8a4be3a01: Status 404 returned error can't find the container with id 0b4e0eb7a9642f0ffe1560b9fd60b5b8f8c78e1f312248318afd29f8a4be3a01 Oct 30 06:26:48.973488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.973447 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:48.973612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.973529 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:48.973612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:48.973558 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:49.051844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.051813 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nnt77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:26:49.052297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.051875 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 06:26:49.228820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.228794 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:26:49.352587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.352522 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-socket\") pod \"9029b845-bbdc-4a56-8776-23996797c555\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " Oct 30 06:26:49.352587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.352591 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-certs\") pod \"9029b845-bbdc-4a56-8776-23996797c555\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " Oct 30 06:26:49.352899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.352868 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9029b845-bbdc-4a56-8776-23996797c555-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9029b845-bbdc-4a56-8776-23996797c555-istiod-ca-cert\") pod \"9029b845-bbdc-4a56-8776-23996797c555\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " Oct 30 06:26:49.353057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.353016 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-istio-token\" (UniqueName: \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-istio-token\") pod \"9029b845-bbdc-4a56-8776-23996797c555\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " Oct 30 06:26:49.353057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.353053 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9029b845-bbdc-4a56-8776-23996797c555-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9029b845-bbdc-4a56-8776-23996797c555-istio-podinfo\") pod \"9029b845-bbdc-4a56-8776-23996797c555\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " Oct 30 06:26:49.353191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.353132 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-data\") pod \"9029b845-bbdc-4a56-8776-23996797c555\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " Oct 30 06:26:49.353251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.353229 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-kube-api-access-bvc57\" (UniqueName: \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-kube-api-access-bvc57\") pod \"9029b845-bbdc-4a56-8776-23996797c555\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " Oct 30 06:26:49.353698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.353670 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-envoy\") pod \"9029b845-bbdc-4a56-8776-23996797c555\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " Oct 30 06:26:49.353846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.353829 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-credential-socket\") pod \"9029b845-bbdc-4a56-8776-23996797c555\" (UID: \"9029b845-bbdc-4a56-8776-23996797c555\") " Oct 30 06:26:49.354049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.354021 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9029b845-bbdc-4a56-8776-23996797c555-istiod-ca-cert" pod "9029b845-bbdc-4a56-8776-23996797c555" (UID: "9029b845-bbdc-4a56-8776-23996797c555"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:49.354358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.354237 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-credential-socket" pod "9029b845-bbdc-4a56-8776-23996797c555" (UID: "9029b845-bbdc-4a56-8776-23996797c555"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:49.354550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.354297 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-socket" pod "9029b845-bbdc-4a56-8776-23996797c555" (UID: "9029b845-bbdc-4a56-8776-23996797c555"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:49.355013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.354988 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:49.355116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.355016 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:49.355116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.355032 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9029b845-bbdc-4a56-8776-23996797c555-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:49.355598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.355571 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-data" pod "9029b845-bbdc-4a56-8776-23996797c555" (UID: "9029b845-bbdc-4a56-8776-23996797c555"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:49.355843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.355785 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-certs" pod "9029b845-bbdc-4a56-8776-23996797c555" (UID: "9029b845-bbdc-4a56-8776-23996797c555"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:49.357609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.357575 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-envoy" pod "9029b845-bbdc-4a56-8776-23996797c555" (UID: "9029b845-bbdc-4a56-8776-23996797c555"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:49.360447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.360418 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9029b845-bbdc-4a56-8776-23996797c555-istio-podinfo" pod "9029b845-bbdc-4a56-8776-23996797c555" (UID: "9029b845-bbdc-4a56-8776-23996797c555"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:26:49.362517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.362484 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-istio-token" pod "9029b845-bbdc-4a56-8776-23996797c555" (UID: "9029b845-bbdc-4a56-8776-23996797c555"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:49.368577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.368540 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-kube-api-access-bvc57" pod "9029b845-bbdc-4a56-8776-23996797c555" (UID: "9029b845-bbdc-4a56-8776-23996797c555"). InnerVolumeSpecName "kube-api-access-bvc57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:49.456375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.456263 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:49.456375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.456295 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9029b845-bbdc-4a56-8776-23996797c555-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:49.456375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.456318 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:49.456375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.456326 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvc57\" (UniqueName: \"kubernetes.io/projected/9029b845-bbdc-4a56-8776-23996797c555-kube-api-access-bvc57\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:49.456375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.456334 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:49.456375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.456342 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9029b845-bbdc-4a56-8776-23996797c555-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:49.482804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.482762 5001 generic.go:356] "Generic (PLEG): container finished" podID="9029b845-bbdc-4a56-8776-23996797c555" containerID="36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32" exitCode=0 Oct 30 06:26:49.482978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.482947 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" Oct 30 06:26:49.483429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.483398 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" event={"ID":"9029b845-bbdc-4a56-8776-23996797c555","Type":"ContainerDied","Data":"36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32"} Oct 30 06:26:49.483568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.483460 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77" event={"ID":"9029b845-bbdc-4a56-8776-23996797c555","Type":"ContainerDied","Data":"de2b8faf246a7cee43b902fb1973d4504822bcca6a5df9ea04370272cf51f6f3"} Oct 30 06:26:49.483568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.483491 5001 scope.go:122] "RemoveContainer" containerID="36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32" Oct 30 06:26:49.485250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.485229 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" event={"ID":"2fc22a5c-aff0-46dc-be59-b438bffb07a9","Type":"ContainerStarted","Data":"7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817"} Oct 30 06:26:49.485371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.485277 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" event={"ID":"2fc22a5c-aff0-46dc-be59-b438bffb07a9","Type":"ContainerStarted","Data":"0b4e0eb7a9642f0ffe1560b9fd60b5b8f8c78e1f312248318afd29f8a4be3a01"} Oct 30 06:26:49.496414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.496389 5001 scope.go:122] "RemoveContainer" containerID="36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32" Oct 30 06:26:49.496689 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:26:49.496670 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32\": container with ID starting with 36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32 not found: ID does not exist" containerID="36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32" Oct 30 06:26:49.496758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.496696 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32"} err="failed to get container status \"36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32\": rpc error: code = NotFound desc = could not find container \"36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32\": container with ID starting with 36ae393d2e7752c19d1ec5d4d8cefb98bd9a2fb367c87b80b3ee1fea57543f32 not found: ID does not exist" Oct 30 06:26:49.537493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.537453 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77"] Oct 30 06:26:49.542049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.542027 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nnt77"] Oct 30 06:26:49.584561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:49.584491 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" podStartSLOduration=2.58447242 podStartE2EDuration="2.58447242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:49.578932283 +0000 UTC m=+2023.574889294" watchObservedRunningTime="2026-10-30 06:26:49.58447242 +0000 UTC m=+2023.580429432" Oct 30 06:26:50.127210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:50.126743 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d687a80-28cf-483b-bbf3-3bc6635fbc8c" path="/var/lib/kubelet/pods/4d687a80-28cf-483b-bbf3-3bc6635fbc8c/volumes" Oct 30 06:26:50.127763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:50.127517 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9029b845-bbdc-4a56-8776-23996797c555" path="/var/lib/kubelet/pods/9029b845-bbdc-4a56-8776-23996797c555/volumes" Oct 30 06:26:50.411019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:50.410934 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:50.416393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:50.416369 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:50.496858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:50.496825 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:50.498011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:50.497991 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:26:51.639804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:51.639772 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c"] Oct 30 06:26:51.640352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:51.640118 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" containerID="cri-o://b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3" gracePeriod=30 Oct 30 06:26:52.399430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.399391 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt"] Oct 30 06:26:52.400495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.400473 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" Oct 30 06:26:52.400495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.400496 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" Oct 30 06:26:52.400628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.400623 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9029b845-bbdc-4a56-8776-23996797c555" containerName="istio-proxy" Oct 30 06:26:52.405969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.405949 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.431839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.431809 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt"] Oct 30 06:26:52.528901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.528863 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.528901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.528898 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.529142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.528915 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.529142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.529056 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.529142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.529113 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.529357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.529164 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.529357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.529190 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn4mx\" (UniqueName: \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-kube-api-access-cn4mx\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.529472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.529369 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.529472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.529388 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.630227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.630184 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.630227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.630233 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.630559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.630345 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.630559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.630372 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.630559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.630397 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.630559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.630471 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.630801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.630574 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.630801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.630613 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.630801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.630649 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn4mx\" (UniqueName: \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-kube-api-access-cn4mx\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.630964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.630805 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.630964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.630859 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.631078 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.631005 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.631165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.631137 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.631511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.631486 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.633714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.633693 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.634068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.634043 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.661341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.661219 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.684380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.664901 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn4mx\" (UniqueName: \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-kube-api-access-cn4mx\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:52.711331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.697279 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:26:52.736454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:52.732582 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:53.223533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:53.223495 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt"] Oct 30 06:26:53.225818 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:53.225786 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5923bc1d_e76f_44df_a3a9_6a3428c52d94.slice/crio-461b2248c0d46fef1016043063c76afd05c3f98e87ac8c77fb5ca9a5c4064be6 WatchSource:0}: Error finding container 461b2248c0d46fef1016043063c76afd05c3f98e87ac8c77fb5ca9a5c4064be6: Status 404 returned error can't find the container with id 461b2248c0d46fef1016043063c76afd05c3f98e87ac8c77fb5ca9a5c4064be6 Oct 30 06:26:53.228969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:53.228923 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:53.229050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:53.229023 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:53.229095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:53.229071 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:53.589833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:53.589726 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" event={"ID":"5923bc1d-e76f-44df-a3a9-6a3428c52d94","Type":"ContainerStarted","Data":"329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3"} Oct 30 06:26:53.590134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:53.590105 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" event={"ID":"5923bc1d-e76f-44df-a3a9-6a3428c52d94","Type":"ContainerStarted","Data":"461b2248c0d46fef1016043063c76afd05c3f98e87ac8c77fb5ca9a5c4064be6"} Oct 30 06:26:53.662895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:53.662829 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podStartSLOduration=1.662807431 podStartE2EDuration="1.662807431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:53.647988382 +0000 UTC m=+2027.643945394" watchObservedRunningTime="2026-10-30 06:26:53.662807431 +0000 UTC m=+2027.658764443" Oct 30 06:26:53.735118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:53.735085 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:26:55.024331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.023646 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 06:26:55.024331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.023729 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 06:26:55.103340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.103135 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn"] Oct 30 06:26:55.200262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.196812 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 06:26:55.216329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.207764 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:26:55.248347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.236631 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn"] Oct 30 06:26:55.270677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.269514 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-fa5a69\"" Oct 30 06:26:55.270677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.269596 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-fa5a69-dockercfg-df6h7\"" Oct 30 06:26:55.354560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.351496 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d8db02-a932-402c-aa37-4e51ac01c609-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn\" (UID: \"c6d8db02-a932-402c-aa37-4e51ac01c609\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 06:26:55.354560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.352146 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2k45\" (UniqueName: \"kubernetes.io/projected/c6d8db02-a932-402c-aa37-4e51ac01c609-kube-api-access-s2k45\") pod \"payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn\" (UID: \"c6d8db02-a932-402c-aa37-4e51ac01c609\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 06:26:55.457792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.442772 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:55.543331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.543222 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:55.561346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.547924 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2k45\" (UniqueName: \"kubernetes.io/projected/c6d8db02-a932-402c-aa37-4e51ac01c609-kube-api-access-s2k45\") pod \"payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn\" (UID: \"c6d8db02-a932-402c-aa37-4e51ac01c609\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 06:26:55.561346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.548270 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d8db02-a932-402c-aa37-4e51ac01c609-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn\" (UID: \"c6d8db02-a932-402c-aa37-4e51ac01c609\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 06:26:55.561811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.561768 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d8db02-a932-402c-aa37-4e51ac01c609-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn\" (UID: \"c6d8db02-a932-402c-aa37-4e51ac01c609\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 06:26:55.604671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.604575 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:55.724542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.724498 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2k45\" (UniqueName: \"kubernetes.io/projected/c6d8db02-a932-402c-aa37-4e51ac01c609-kube-api-access-s2k45\") pod \"payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn\" (UID: \"c6d8db02-a932-402c-aa37-4e51ac01c609\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 06:26:55.747383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.747346 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 06:26:55.747603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.747407 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 06:26:55.919727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:55.919627 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 06:26:56.481658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:56.481550 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:56.483752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:56.483714 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:26:56.544453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:56.544413 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:56.545903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:56.545869 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:26:56.872362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:56.850437 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 06:26:56.872362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:56.850508 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 06:26:57.533634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.533498 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:57.578375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.577055 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 06:26:57.609848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.608097 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:57.652704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.645091 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:57.662809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.661574 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z"] Oct 30 06:26:57.662809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.661821 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" containerID="cri-o://99a957f6bdca6072b90adf2cc6a837466a8fb65f60154d49667b7449a7a90af3" gracePeriod=30 Oct 30 06:26:57.672645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.672616 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn"] Oct 30 06:26:57.681412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.677987 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg"] Oct 30 06:26:57.685332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.685228 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:26:57.702374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.688049 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg"] Oct 30 06:26:57.702374 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:57.692177 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d8db02_a932_402c_aa37_4e51ac01c609.slice/crio-22a229ea62330dc56d2f94c8627a7b891f954684583e1516c871982307060881 WatchSource:0}: Error finding container 22a229ea62330dc56d2f94c8627a7b891f954684583e1516c871982307060881: Status 404 returned error can't find the container with id 22a229ea62330dc56d2f94c8627a7b891f954684583e1516c871982307060881 Oct 30 06:26:57.715362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.713055 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 06:26:57.731056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.731020 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:26:57.731243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.731154 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:26:57.731243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.731205 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:26:57.731454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.731286 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:26:57.747891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.738067 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 06:26:57.747891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.738134 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 06:26:57.778597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.772224 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:26:57.778597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.778125 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:26:57.778597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.778180 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2nwn\" (UniqueName: \"kubernetes.io/projected/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-kube-api-access-s2nwn\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:26:57.778597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.778500 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:57.795578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.795529 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:57.889472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.886486 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:26:57.889472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.886538 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2nwn\" (UniqueName: \"kubernetes.io/projected/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-kube-api-access-s2nwn\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:26:57.889472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.886624 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:26:57.889472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.887864 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:26:57.920353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:57.918795 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:26:58.139125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.132688 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2nwn\" (UniqueName: \"kubernetes.io/projected/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-kube-api-access-s2nwn\") pod \"maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:26:58.152623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.150645 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:26:58.314084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.314033 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:58.425492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.425463 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:26:58.429588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.429557 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 06:26:58.429750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.429607 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:26:58.532377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.532287 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-kube-api-access-br5jt\" (UniqueName: \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-kube-api-access-br5jt\") pod \"446c1aec-14d5-468e-887d-39c6579f389c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " Oct 30 06:26:58.532377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.532361 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-istio-token\" (UniqueName: \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-istio-token\") pod \"446c1aec-14d5-468e-887d-39c6579f389c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " Oct 30 06:26:58.532628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.532423 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/446c1aec-14d5-468e-887d-39c6579f389c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446c1aec-14d5-468e-887d-39c6579f389c-istio-podinfo\") pod \"446c1aec-14d5-468e-887d-39c6579f389c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " Oct 30 06:26:58.532628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.532460 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-credential-socket\") pod \"446c1aec-14d5-468e-887d-39c6579f389c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " Oct 30 06:26:58.532628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.532506 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-envoy\") pod \"446c1aec-14d5-468e-887d-39c6579f389c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " Oct 30 06:26:58.532817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.532655 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-socket\") pod \"446c1aec-14d5-468e-887d-39c6579f389c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " Oct 30 06:26:58.532817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.532701 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-certs\") pod \"446c1aec-14d5-468e-887d-39c6579f389c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " Oct 30 06:26:58.532817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.532748 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/446c1aec-14d5-468e-887d-39c6579f389c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446c1aec-14d5-468e-887d-39c6579f389c-istiod-ca-cert\") pod \"446c1aec-14d5-468e-887d-39c6579f389c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " Oct 30 06:26:58.532979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.532943 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-data\") pod \"446c1aec-14d5-468e-887d-39c6579f389c\" (UID: \"446c1aec-14d5-468e-887d-39c6579f389c\") " Oct 30 06:26:58.534909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.534014 5001 patch_prober.go:29] interesting pod/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:26:58.534909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.534065 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:26:58.534909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.534616 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-credential-socket" pod "446c1aec-14d5-468e-887d-39c6579f389c" (UID: "446c1aec-14d5-468e-887d-39c6579f389c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:58.534909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.534631 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-certs" pod "446c1aec-14d5-468e-887d-39c6579f389c" (UID: "446c1aec-14d5-468e-887d-39c6579f389c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:58.538181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.538114 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-kube-api-access-br5jt" pod "446c1aec-14d5-468e-887d-39c6579f389c" (UID: "446c1aec-14d5-468e-887d-39c6579f389c"). InnerVolumeSpecName "kube-api-access-br5jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:58.538493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.538471 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446c1aec-14d5-468e-887d-39c6579f389c-istiod-ca-cert" pod "446c1aec-14d5-468e-887d-39c6579f389c" (UID: "446c1aec-14d5-468e-887d-39c6579f389c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:58.538915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.538723 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-data" pod "446c1aec-14d5-468e-887d-39c6579f389c" (UID: "446c1aec-14d5-468e-887d-39c6579f389c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:58.539669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.539609 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-socket" pod "446c1aec-14d5-468e-887d-39c6579f389c" (UID: "446c1aec-14d5-468e-887d-39c6579f389c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:58.539885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.539840 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-envoy" pod "446c1aec-14d5-468e-887d-39c6579f389c" (UID: "446c1aec-14d5-468e-887d-39c6579f389c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:58.539885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.539859 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-istio-token" pod "446c1aec-14d5-468e-887d-39c6579f389c" (UID: "446c1aec-14d5-468e-887d-39c6579f389c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:58.542316 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.542277 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/446c1aec-14d5-468e-887d-39c6579f389c-istio-podinfo" pod "446c1aec-14d5-468e-887d-39c6579f389c" (UID: "446c1aec-14d5-468e-887d-39c6579f389c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:26:58.635710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.635599 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:58.635710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.635644 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br5jt\" (UniqueName: \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-kube-api-access-br5jt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:58.635710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.635660 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446c1aec-14d5-468e-887d-39c6579f389c-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:58.635710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.635674 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446c1aec-14d5-468e-887d-39c6579f389c-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:58.636200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.635688 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:58.636200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.636160 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:58.636200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.636177 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:58.636485 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.636444 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446c1aec-14d5-468e-887d-39c6579f389c-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:58.636485 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.636467 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446c1aec-14d5-468e-887d-39c6579f389c-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:26:58.747782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.747747 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 06:26:58.747977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.747808 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 06:26:58.825527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.824570 5001 generic.go:356] "Generic (PLEG): container finished" podID="446c1aec-14d5-468e-887d-39c6579f389c" containerID="b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3" exitCode=0 Oct 30 06:26:58.825527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.824717 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" event={"ID":"446c1aec-14d5-468e-887d-39c6579f389c","Type":"ContainerDied","Data":"b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3"} Oct 30 06:26:58.825527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.824760 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" event={"ID":"446c1aec-14d5-468e-887d-39c6579f389c","Type":"ContainerDied","Data":"7952236c34dc1bce96f68e7fe11eceae26d58b4135f1efdbc450a7fcd68a6d5a"} Oct 30 06:26:58.825527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.824781 5001 scope.go:122] "RemoveContainer" containerID="b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3" Oct 30 06:26:58.825527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.824902 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c" Oct 30 06:26:58.837766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.836359 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" event={"ID":"c6d8db02-a932-402c-aa37-4e51ac01c609","Type":"ContainerStarted","Data":"168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28"} Oct 30 06:26:58.837766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.836408 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" event={"ID":"c6d8db02-a932-402c-aa37-4e51ac01c609","Type":"ContainerStarted","Data":"22a229ea62330dc56d2f94c8627a7b891f954684583e1516c871982307060881"} Oct 30 06:26:58.837766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.836533 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 06:26:58.845444 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.845419 5001 scope.go:122] "RemoveContainer" containerID="51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343" Oct 30 06:26:58.925146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.925101 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c"] Oct 30 06:26:58.937689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.936679 5001 scope.go:122] "RemoveContainer" containerID="b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3" Oct 30 06:26:58.938201 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:26:58.938141 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3\": container with ID starting with b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3 not found: ID does not exist" containerID="b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3" Oct 30 06:26:58.938201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.938181 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3"} err="failed to get container status \"b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3\": rpc error: code = NotFound desc = could not find container \"b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3\": container with ID starting with b836a3b4c356173bc76d824dc8743e526595d966361781997ff37f3611d864d3 not found: ID does not exist" Oct 30 06:26:58.938447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.938206 5001 scope.go:122] "RemoveContainer" containerID="51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343" Oct 30 06:26:58.938867 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:26:58.938781 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343\": container with ID starting with 51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343 not found: ID does not exist" containerID="51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343" Oct 30 06:26:58.938867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.938826 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343"} err="failed to get container status \"51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343\": rpc error: code = NotFound desc = could not find container \"51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343\": container with ID starting with 51a8e34204fa65d83cb55036223a839348f8b78f5dcb2b53d3f2e365877cf343 not found: ID does not exist" Oct 30 06:26:58.940223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:58.940199 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-5a6371-openshift-default-786c45cf4f-t224c"] Oct 30 06:26:59.027956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.012457 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" podStartSLOduration=5.012433162 podStartE2EDuration="5.012433162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:58.997697189 +0000 UTC m=+2032.993654200" watchObservedRunningTime="2026-10-30 06:26:59.012433162 +0000 UTC m=+2033.008390173" Oct 30 06:26:59.028500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.028435 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg"] Oct 30 06:26:59.028986 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:26:59.028940 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3555d2d8_8bfb_4d03_8f12_4280b46dcc8f.slice/crio-a1f5a045e7040abb0a1d9ed52c95dbd043b0aeeccb826ef6a567edb691e04177 WatchSource:0}: Error finding container a1f5a045e7040abb0a1d9ed52c95dbd043b0aeeccb826ef6a567edb691e04177: Status 404 returned error can't find the container with id a1f5a045e7040abb0a1d9ed52c95dbd043b0aeeccb826ef6a567edb691e04177 Oct 30 06:26:59.442141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.442041 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6"] Oct 30 06:26:59.443544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.443433 5001 cpu_manager.go:414] "containerMap: removing container" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" Oct 30 06:26:59.443716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.443699 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" Oct 30 06:26:59.443821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.443809 5001 cpu_manager.go:414] "containerMap: removing container" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" Oct 30 06:26:59.443914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.443894 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" Oct 30 06:26:59.444254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.444236 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" Oct 30 06:26:59.444391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.444379 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="446c1aec-14d5-468e-887d-39c6579f389c" containerName="istio-proxy" Oct 30 06:26:59.475006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.474978 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:26:59.514330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.489797 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791a33a9-9513-4bd6-bd64-0c37766184df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6\" (UID: \"791a33a9-9513-4bd6-bd64-0c37766184df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:26:59.523719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.520927 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdrfv\" (UniqueName: \"kubernetes.io/projected/791a33a9-9513-4bd6-bd64-0c37766184df-kube-api-access-jdrfv\") pod \"payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6\" (UID: \"791a33a9-9513-4bd6-bd64-0c37766184df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:26:59.523719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.501703 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6"] Oct 30 06:26:59.523719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.522756 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-4f3973\"" Oct 30 06:26:59.524623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.524590 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd"] Oct 30 06:26:59.529260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.529240 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:26:59.532325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.532286 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd"] Oct 30 06:26:59.546788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.546772 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-4f3973-dockercfg-ssb22\"" Oct 30 06:26:59.622343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.622272 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791a33a9-9513-4bd6-bd64-0c37766184df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6\" (UID: \"791a33a9-9513-4bd6-bd64-0c37766184df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:26:59.623644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.623606 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c77805f-d7de-44d8-8a27-3c910cb60b4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd\" (UID: \"1c77805f-d7de-44d8-8a27-3c910cb60b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:26:59.623792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.623708 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdrfv\" (UniqueName: \"kubernetes.io/projected/791a33a9-9513-4bd6-bd64-0c37766184df-kube-api-access-jdrfv\") pod \"payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6\" (UID: \"791a33a9-9513-4bd6-bd64-0c37766184df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:26:59.623870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.623827 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwtt\" (UniqueName: \"kubernetes.io/projected/1c77805f-d7de-44d8-8a27-3c910cb60b4e-kube-api-access-6gwtt\") pod \"payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd\" (UID: \"1c77805f-d7de-44d8-8a27-3c910cb60b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:26:59.623994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.623415 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791a33a9-9513-4bd6-bd64-0c37766184df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6\" (UID: \"791a33a9-9513-4bd6-bd64-0c37766184df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:26:59.725382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.723280 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdrfv\" (UniqueName: \"kubernetes.io/projected/791a33a9-9513-4bd6-bd64-0c37766184df-kube-api-access-jdrfv\") pod \"payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6\" (UID: \"791a33a9-9513-4bd6-bd64-0c37766184df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:26:59.728883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.728844 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c77805f-d7de-44d8-8a27-3c910cb60b4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd\" (UID: \"1c77805f-d7de-44d8-8a27-3c910cb60b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:26:59.729091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.729067 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c77805f-d7de-44d8-8a27-3c910cb60b4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd\" (UID: \"1c77805f-d7de-44d8-8a27-3c910cb60b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:26:59.729352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.729287 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwtt\" (UniqueName: \"kubernetes.io/projected/1c77805f-d7de-44d8-8a27-3c910cb60b4e-kube-api-access-6gwtt\") pod \"payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd\" (UID: \"1c77805f-d7de-44d8-8a27-3c910cb60b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:26:59.735611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.735585 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 06:26:59.735727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.735641 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 06:26:59.780205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.780165 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwtt\" (UniqueName: \"kubernetes.io/projected/1c77805f-d7de-44d8-8a27-3c910cb60b4e-kube-api-access-6gwtt\") pod \"payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd\" (UID: \"1c77805f-d7de-44d8-8a27-3c910cb60b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:26:59.843292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.843263 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:26:59.848643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.848613 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" event={"ID":"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f","Type":"ContainerStarted","Data":"720e17bca026c9c3ed536effcab6f94159f2bad12fa10d7604a36662aaeca6ef"} Oct 30 06:26:59.848811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.848667 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" event={"ID":"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f","Type":"ContainerStarted","Data":"a1f5a045e7040abb0a1d9ed52c95dbd043b0aeeccb826ef6a567edb691e04177"} Oct 30 06:26:59.849008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.848988 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:26:59.973476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:26:59.966763 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" podStartSLOduration=2.9667297919999998 podStartE2EDuration="2.966729792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:59.892376179 +0000 UTC m=+2033.888333190" watchObservedRunningTime="2026-10-30 06:26:59.966729792 +0000 UTC m=+2033.962686804" Oct 30 06:27:00.140925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:00.140890 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="446c1aec-14d5-468e-887d-39c6579f389c" path="/var/lib/kubelet/pods/446c1aec-14d5-468e-887d-39c6579f389c/volumes" Oct 30 06:27:00.526850 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:27:00.526805 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791a33a9_9513_4bd6_bd64_0c37766184df.slice/crio-65e31f74be11be3bdec3ae855b0a1f4176b127daa78220594b8f520131084582 WatchSource:0}: Error finding container 65e31f74be11be3bdec3ae855b0a1f4176b127daa78220594b8f520131084582: Status 404 returned error can't find the container with id 65e31f74be11be3bdec3ae855b0a1f4176b127daa78220594b8f520131084582 Oct 30 06:27:00.530136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:00.530101 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6"] Oct 30 06:27:00.735294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:00.735271 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 06:27:00.735634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:00.735359 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 06:27:00.831851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:00.831547 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd"] Oct 30 06:27:00.936569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:00.936379 5001 generic.go:356] "Generic (PLEG): container finished" podID="3555d2d8-8bfb-4d03-8f12-4280b46dcc8f" containerID="720e17bca026c9c3ed536effcab6f94159f2bad12fa10d7604a36662aaeca6ef" exitCode=0 Oct 30 06:27:00.936569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:00.936486 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" event={"ID":"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f","Type":"ContainerDied","Data":"720e17bca026c9c3ed536effcab6f94159f2bad12fa10d7604a36662aaeca6ef"} Oct 30 06:27:00.939332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:00.938837 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" event={"ID":"791a33a9-9513-4bd6-bd64-0c37766184df","Type":"ContainerStarted","Data":"ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40"} Oct 30 06:27:00.939332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:00.938931 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" event={"ID":"791a33a9-9513-4bd6-bd64-0c37766184df","Type":"ContainerStarted","Data":"65e31f74be11be3bdec3ae855b0a1f4176b127daa78220594b8f520131084582"} Oct 30 06:27:00.939332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:00.939091 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:27:00.940792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:00.940772 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" event={"ID":"1c77805f-d7de-44d8-8a27-3c910cb60b4e","Type":"ContainerStarted","Data":"aad76a0677452648dc29adc50cca7a29ae5fac80a19a6fca85210d318c420e3c"} Oct 30 06:27:01.117331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:01.117190 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" podStartSLOduration=2.11716656 podStartE2EDuration="2.11716656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:01.073086611 +0000 UTC m=+2035.069043621" watchObservedRunningTime="2026-10-30 06:27:01.11716656 +0000 UTC m=+2035.113123570" Oct 30 06:27:01.736388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:01.735575 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 06:27:01.736388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:01.735633 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 06:27:01.955439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:01.955028 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" event={"ID":"1c77805f-d7de-44d8-8a27-3c910cb60b4e","Type":"ContainerStarted","Data":"ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721"} Oct 30 06:27:01.955721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:01.955701 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:27:02.017399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.001175 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" podStartSLOduration=3.001157635 podStartE2EDuration="3.001157635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:02.000900929 +0000 UTC m=+2035.996857967" watchObservedRunningTime="2026-10-30 06:27:02.001157635 +0000 UTC m=+2035.997114668" Oct 30 06:27:02.329128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.329103 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:27:02.388967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.388931 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-kube-api-access-s2nwn\" (UniqueName: \"kubernetes.io/projected/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-kube-api-access-s2nwn\") pod \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " Oct 30 06:27:02.389157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.389045 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-xks-ca\") pod \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " Oct 30 06:27:02.389224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.389189 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-openshift-ca\") pod \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\" (UID: \"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f\") " Oct 30 06:27:02.390176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.390142 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-openshift-ca" pod "3555d2d8-8bfb-4d03-8f12-4280b46dcc8f" (UID: "3555d2d8-8bfb-4d03-8f12-4280b46dcc8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:02.392360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.392319 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-kube-api-access-s2nwn" pod "3555d2d8-8bfb-4d03-8f12-4280b46dcc8f" (UID: "3555d2d8-8bfb-4d03-8f12-4280b46dcc8f"). InnerVolumeSpecName "kube-api-access-s2nwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:02.393058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.393025 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-xks-ca" pod "3555d2d8-8bfb-4d03-8f12-4280b46dcc8f" (UID: "3555d2d8-8bfb-4d03-8f12-4280b46dcc8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:02.491111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.491074 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:02.491111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.491111 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2nwn\" (UniqueName: \"kubernetes.io/projected/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-kube-api-access-s2nwn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:02.491417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.491126 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:02.735159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.735126 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 06:27:02.735356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:02.735183 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 06:27:03.124289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.124242 5001 generic.go:356] "Generic (PLEG): container finished" podID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerID="99a957f6bdca6072b90adf2cc6a837466a8fb65f60154d49667b7449a7a90af3" exitCode=0 Oct 30 06:27:03.124769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.124529 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" event={"ID":"03ed5ca9-d682-4636-a96b-e7d2b84b8455","Type":"ContainerDied","Data":"99a957f6bdca6072b90adf2cc6a837466a8fb65f60154d49667b7449a7a90af3"} Oct 30 06:27:03.124769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.124616 5001 scope.go:122] "RemoveContainer" containerID="e1d702587347d396cebe122245345bb5a55542840cc54958553a06665dff9941" Oct 30 06:27:03.129909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.129876 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" Oct 30 06:27:03.133491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.132899 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg" event={"ID":"3555d2d8-8bfb-4d03-8f12-4280b46dcc8f","Type":"ContainerDied","Data":"a1f5a045e7040abb0a1d9ed52c95dbd043b0aeeccb826ef6a567edb691e04177"} Oct 30 06:27:03.133491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.133058 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f5a045e7040abb0a1d9ed52c95dbd043b0aeeccb826ef6a567edb691e04177" Oct 30 06:27:03.151236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.151210 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:27:03.239730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.239685 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-envoy\") pod \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " Oct 30 06:27:03.239953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.239828 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-token\" (UniqueName: \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-token\") pod \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " Oct 30 06:27:03.239953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.239884 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-credential-socket\") pod \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " Oct 30 06:27:03.239953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.239941 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-kube-api-access-nh888\" (UniqueName: \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-kube-api-access-nh888\") pod \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " Oct 30 06:27:03.240157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.240014 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-data\") pod \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " Oct 30 06:27:03.240157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.240092 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istiod-ca-cert\") pod \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " Oct 30 06:27:03.240274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.240165 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-certs\") pod \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " Oct 30 06:27:03.240274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.240201 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-socket\") pod \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " Oct 30 06:27:03.240274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.240237 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-podinfo\") pod \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\" (UID: \"03ed5ca9-d682-4636-a96b-e7d2b84b8455\") " Oct 30 06:27:03.241149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.241083 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-data" pod "03ed5ca9-d682-4636-a96b-e7d2b84b8455" (UID: "03ed5ca9-d682-4636-a96b-e7d2b84b8455"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:03.241294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.241131 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istiod-ca-cert" pod "03ed5ca9-d682-4636-a96b-e7d2b84b8455" (UID: "03ed5ca9-d682-4636-a96b-e7d2b84b8455"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:03.241714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.241686 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-socket" pod "03ed5ca9-d682-4636-a96b-e7d2b84b8455" (UID: "03ed5ca9-d682-4636-a96b-e7d2b84b8455"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:03.242042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.241993 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-certs" pod "03ed5ca9-d682-4636-a96b-e7d2b84b8455" (UID: "03ed5ca9-d682-4636-a96b-e7d2b84b8455"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:03.242145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.242099 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-credential-socket" pod "03ed5ca9-d682-4636-a96b-e7d2b84b8455" (UID: "03ed5ca9-d682-4636-a96b-e7d2b84b8455"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:03.242447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.242424 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:03.242541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.242489 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:03.242541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.242506 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:03.242662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.242519 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:03.242662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.242650 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:03.244924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.244880 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-podinfo" pod "03ed5ca9-d682-4636-a96b-e7d2b84b8455" (UID: "03ed5ca9-d682-4636-a96b-e7d2b84b8455"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:27:03.245901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.245879 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-token" pod "03ed5ca9-d682-4636-a96b-e7d2b84b8455" (UID: "03ed5ca9-d682-4636-a96b-e7d2b84b8455"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:03.246890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.246855 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-envoy" pod "03ed5ca9-d682-4636-a96b-e7d2b84b8455" (UID: "03ed5ca9-d682-4636-a96b-e7d2b84b8455"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:03.247274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.247251 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-kube-api-access-nh888" pod "03ed5ca9-d682-4636-a96b-e7d2b84b8455" (UID: "03ed5ca9-d682-4636-a96b-e7d2b84b8455"). InnerVolumeSpecName "kube-api-access-nh888". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:03.344108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.344065 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:03.344108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.344107 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:03.344343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.344124 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh888\" (UniqueName: \"kubernetes.io/projected/03ed5ca9-d682-4636-a96b-e7d2b84b8455-kube-api-access-nh888\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:03.344343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.344138 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ed5ca9-d682-4636-a96b-e7d2b84b8455-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:03.735426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.735383 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 06:27:03.735609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:03.735448 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 06:27:04.153729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:04.153685 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" event={"ID":"03ed5ca9-d682-4636-a96b-e7d2b84b8455","Type":"ContainerDied","Data":"9a010b9c52a6721ab862f362862cd92ccdf2e5d668c80f959dbdb071428d5bfa"} Oct 30 06:27:04.154112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:04.153762 5001 scope.go:122] "RemoveContainer" containerID="99a957f6bdca6072b90adf2cc6a837466a8fb65f60154d49667b7449a7a90af3" Oct 30 06:27:04.154112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:04.153897 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z" Oct 30 06:27:04.277049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:04.277014 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z"] Oct 30 06:27:04.286423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:04.286388 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f5e55f-openshift-default-698f5b89b5-pf77z"] Oct 30 06:27:04.740968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:04.740937 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:27:04.741550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:04.741520 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:27:04.742538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:04.742516 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:27:05.783262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:05.783219 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:27:05.783956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:05.783918 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:27:06.121125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:06.121092 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" path="/var/lib/kubelet/pods/03ed5ca9-d682-4636-a96b-e7d2b84b8455/volumes" Oct 30 06:27:07.209387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:07.209353 5001 generic.go:356] "Generic (PLEG): container finished" podID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerID="7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c" exitCode=0 Oct 30 06:27:07.209857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:07.209415 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" event={"ID":"4ad31106-b066-48b3-9bc4-a8e442cb719e","Type":"ContainerDied","Data":"7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c"} Oct 30 06:27:07.209937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:07.209916 5001 scope.go:122] "RemoveContainer" containerID="7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c" Oct 30 06:27:07.211554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:07.211528 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:07.211626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:07.211594 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:07.211673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:07.211624 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:08.221170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:08.221131 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" event={"ID":"4ad31106-b066-48b3-9bc4-a8e442cb719e","Type":"ContainerStarted","Data":"a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada"} Oct 30 06:27:08.355327 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:27:08.354153 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:27:08.355948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:08.355865 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:27:08.356560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:08.356189 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 06:27:08.356560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:08.356239 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 06:27:08.879110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:08.879067 5001 scope.go:122] "RemoveContainer" containerID="0620690edad1eee9601fdb79c8db4c1f268557c745585737c1548df79152e85d" Oct 30 06:27:08.894811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:08.894785 5001 scope.go:122] "RemoveContainer" containerID="baa0e95a0663b26bc4747fecd65af0ea8e2b3a7ff68a1e53c29b94743e8261e0" Oct 30 06:27:08.909566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:08.909227 5001 scope.go:122] "RemoveContainer" containerID="32e849a8114adad7db011e86447967e144c6bd5428ce3f53cf4ea7b9f392a688" Oct 30 06:27:08.936238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:08.936216 5001 scope.go:122] "RemoveContainer" containerID="c0436674e950e530290a77401cf4594858d0e23328b107f8f3b27402e245dedc" Oct 30 06:27:09.344132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:09.344031 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 06:27:09.344132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:09.344105 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 06:27:09.875812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:09.874497 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 06:27:10.344129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:10.344044 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 06:27:10.344129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:10.344097 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 06:27:11.343723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:11.343687 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 06:27:11.343919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:11.343739 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 06:27:11.981543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:11.981506 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:27:12.343805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:12.343730 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 06:27:12.344036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:12.344004 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 06:27:13.142096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:13.142066 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:27:13.344095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:13.344021 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 06:27:13.344412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:13.344387 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 06:27:14.347680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:14.347652 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:27:14.348093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:14.348039 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:27:14.348755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:14.348732 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 06:27:17.924210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.924167 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx"] Oct 30 06:27:17.925292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.925263 5001 cpu_manager.go:414] "containerMap: removing container" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" Oct 30 06:27:17.925292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.925288 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" Oct 30 06:27:17.925532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.925377 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3555d2d8-8bfb-4d03-8f12-4280b46dcc8f" containerName="revoke-keys" Oct 30 06:27:17.925532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.925387 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3555d2d8-8bfb-4d03-8f12-4280b46dcc8f" containerName="revoke-keys" Oct 30 06:27:17.925532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.925421 5001 cpu_manager.go:414] "containerMap: removing container" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" Oct 30 06:27:17.925532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.925431 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" Oct 30 06:27:17.925765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.925610 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" Oct 30 06:27:17.925765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.925633 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3555d2d8-8bfb-4d03-8f12-4280b46dcc8f" containerName="revoke-keys" Oct 30 06:27:17.925765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.925647 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ed5ca9-d682-4636-a96b-e7d2b84b8455" containerName="istio-proxy" Oct 30 06:27:17.933561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.933534 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:17.947920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.947849 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-30fbb6-openshift-default-dockercfg-rm4rt\"" Oct 30 06:27:17.950949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:17.950892 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx"] Oct 30 06:27:18.073207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.073161 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4549b201-684a-49f1-b871-4c0866e27bbb-istio-podinfo\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.073607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.073584 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-certs\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.073819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.073803 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-envoy\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.077330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.074046 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjgv2\" (UniqueName: \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-kube-api-access-gjgv2\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.077592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.077571 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-data\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.077792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.077776 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-socket\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.077996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.077979 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-istio-token\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.095332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.078140 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4549b201-684a-49f1-b871-4c0866e27bbb-istiod-ca-cert\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.095690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.095658 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-credential-socket\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.197526 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-certs\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.197581 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-envoy\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.197677 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjgv2\" (UniqueName: \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-kube-api-access-gjgv2\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.197708 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-data\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.197761 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-socket\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.197822 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-istio-token\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.197845 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4549b201-684a-49f1-b871-4c0866e27bbb-istiod-ca-cert\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.197884 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-credential-socket\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.198650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.197999 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4549b201-684a-49f1-b871-4c0866e27bbb-istio-podinfo\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.199677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.199650 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-data\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.200944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.200917 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-certs\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.214587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.202088 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-socket\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.216039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.216006 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4549b201-684a-49f1-b871-4c0866e27bbb-istiod-ca-cert\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.217898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.217189 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-credential-socket\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.231001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.230777 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-istio-token\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.262668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.252415 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4549b201-684a-49f1-b871-4c0866e27bbb-istio-podinfo\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.262668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.259634 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjgv2\" (UniqueName: \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-kube-api-access-gjgv2\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.269662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.269614 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-envoy\") pod \"e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.557442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.557343 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:18.906282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.906251 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx"] Oct 30 06:27:18.907484 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:27:18.907458 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4549b201_684a_49f1_b871_4c0866e27bbb.slice/crio-fe73c1492a88b04d23df40ee6ac981e490824f793330941766264bc2afac2741 WatchSource:0}: Error finding container fe73c1492a88b04d23df40ee6ac981e490824f793330941766264bc2afac2741: Status 404 returned error can't find the container with id fe73c1492a88b04d23df40ee6ac981e490824f793330941766264bc2afac2741 Oct 30 06:27:18.910096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.910059 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:18.910205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.910130 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:18.910205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:18.910160 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:19.414277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.409433 5001 generic.go:356] "Generic (PLEG): container finished" podID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerID="999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a" exitCode=0 Oct 30 06:27:19.414277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.409903 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" event={"ID":"6d6a40bf-862a-49e7-9fd5-a75582747c0d","Type":"ContainerDied","Data":"999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a"} Oct 30 06:27:19.414277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.410592 5001 scope.go:122] "RemoveContainer" containerID="999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a" Oct 30 06:27:19.414277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.413482 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:19.414277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.413573 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:19.414277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.413618 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:19.436254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.436221 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" event={"ID":"4549b201-684a-49f1-b871-4c0866e27bbb","Type":"ContainerStarted","Data":"f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7"} Oct 30 06:27:19.436578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.436553 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" event={"ID":"4549b201-684a-49f1-b871-4c0866e27bbb","Type":"ContainerStarted","Data":"fe73c1492a88b04d23df40ee6ac981e490824f793330941766264bc2afac2741"} Oct 30 06:27:19.511500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.511451 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" podStartSLOduration=2.5114342130000002 podStartE2EDuration="2.511434213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:19.507721202 +0000 UTC m=+2053.503678214" watchObservedRunningTime="2026-10-30 06:27:19.511434213 +0000 UTC m=+2053.507391222" Oct 30 06:27:19.558228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.558141 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:19.564898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.564570 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:19.881332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:19.880889 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:27:20.524330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:20.524110 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" event={"ID":"6d6a40bf-862a-49e7-9fd5-a75582747c0d","Type":"ContainerStarted","Data":"de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064"} Oct 30 06:27:20.524330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:20.524174 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:20.543330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:20.538888 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:27:21.123407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:21.123359 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:27:21.123635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:21.123509 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:27:21.123635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:21.123576 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 06:27:22.137333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:22.136869 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:27:22.137333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:22.136941 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 06:27:23.125769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:23.125727 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:27:23.126077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:23.126045 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 06:27:24.300412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.286498 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:27:24.300412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.286643 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 06:27:24.638270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.638228 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 06:27:24.670330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.669899 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:27:24.706445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.706395 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:27:24.733501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.733462 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw"] Oct 30 06:27:24.806952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.797649 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw"] Oct 30 06:27:24.806952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.797776 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 06:27:24.837343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.836699 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-30fbb6-dockercfg-75j5s\"" Oct 30 06:27:24.870036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.869939 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-30fbb6\"" Oct 30 06:27:24.941937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.941833 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcbsn\" (UniqueName: \"kubernetes.io/projected/0b05b248-f62b-466f-b923-311b6d99c48e-kube-api-access-rcbsn\") pod \"payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw\" (UID: \"0b05b248-f62b-466f-b923-311b6d99c48e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 06:27:24.942376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:24.942349 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b05b248-f62b-466f-b923-311b6d99c48e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw\" (UID: \"0b05b248-f62b-466f-b923-311b6d99c48e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 06:27:25.043469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:25.043429 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcbsn\" (UniqueName: \"kubernetes.io/projected/0b05b248-f62b-466f-b923-311b6d99c48e-kube-api-access-rcbsn\") pod \"payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw\" (UID: \"0b05b248-f62b-466f-b923-311b6d99c48e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 06:27:25.044261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:25.044230 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b05b248-f62b-466f-b923-311b6d99c48e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw\" (UID: \"0b05b248-f62b-466f-b923-311b6d99c48e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 06:27:25.045216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:25.045185 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b05b248-f62b-466f-b923-311b6d99c48e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw\" (UID: \"0b05b248-f62b-466f-b923-311b6d99c48e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 06:27:25.092070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:25.092031 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcbsn\" (UniqueName: \"kubernetes.io/projected/0b05b248-f62b-466f-b923-311b6d99c48e-kube-api-access-rcbsn\") pod \"payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw\" (UID: \"0b05b248-f62b-466f-b923-311b6d99c48e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 06:27:25.129784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:25.129743 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:27:25.130072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:25.130043 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 06:27:25.221649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:25.221550 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 06:27:26.224399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:26.224364 5001 patch_prober.go:29] interesting pod/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:27:26.225013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:26.224978 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 06:27:26.451057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:26.450810 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw"] Oct 30 06:27:27.129006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:27.128975 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:27:27.129529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:27.129512 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:27:27.130346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:27.130324 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:27:27.310426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:27.310391 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" event={"ID":"0b05b248-f62b-466f-b923-311b6d99c48e","Type":"ContainerStarted","Data":"550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0"} Oct 30 06:27:27.310985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:27.310954 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 06:27:27.311127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:27.310998 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" event={"ID":"0b05b248-f62b-466f-b923-311b6d99c48e","Type":"ContainerStarted","Data":"06b60a36aa5673a0a0db464fccd5d5bfb9da84ddd74f9ce5360bc64233b1630e"} Oct 30 06:27:27.373640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:27.366159 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" podStartSLOduration=4.366135184 podStartE2EDuration="4.366135184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:27.349856742 +0000 UTC m=+2061.345813752" watchObservedRunningTime="2026-10-30 06:27:27.366135184 +0000 UTC m=+2061.362092198" Oct 30 06:27:27.888435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:27.888363 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg"] Oct 30 06:27:27.899159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:27.899123 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ec27ed-p62cg"] Oct 30 06:27:28.121247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:28.121207 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3555d2d8-8bfb-4d03-8f12-4280b46dcc8f" path="/var/lib/kubelet/pods/3555d2d8-8bfb-4d03-8f12-4280b46dcc8f/volumes" Oct 30 06:27:31.581513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:31.581462 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:27:31.582019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:31.581537 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:27:32.196030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.195996 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6"] Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.221906 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6"] Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.222038 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.265164 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-868b1f-openshift-default-dockercfg-qdrc7\"" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.349529 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-envoy\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.349589 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-token\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.349670 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-data\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.349722 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istiod-ca-cert\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.349766 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668nv\" (UniqueName: \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-kube-api-access-668nv\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.349993 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-credential-socket\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.350025 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-certs\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.350094 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-socket\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.350129 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-podinfo\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.463683 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-data\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.463756 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istiod-ca-cert\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.463800 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668nv\" (UniqueName: \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-kube-api-access-668nv\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.463930 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-credential-socket\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.463956 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-certs\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.464030 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-socket\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.464064 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-podinfo\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.464180 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-envoy\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.464219 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-token\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.464737 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-credential-socket\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.464964 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-socket\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.465696 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-certs\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.466404 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-data\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.574028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.467992 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istiod-ca-cert\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.598066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.588438 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-token\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.598066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.589735 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668nv\" (UniqueName: \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-kube-api-access-668nv\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.598066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.589766 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-envoy\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.732841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.599588 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-podinfo\") pod \"e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:32.856895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:32.856862 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:33.194183 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:27:33.194152 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc741b57d_5029_4bfe_ab30_f58e5e45eb2c.slice/crio-c637dbbb344dcc6186cb00dfa61cf4f4078bb8ebf8993b48d5114eed57c41698 WatchSource:0}: Error finding container c637dbbb344dcc6186cb00dfa61cf4f4078bb8ebf8993b48d5114eed57c41698: Status 404 returned error can't find the container with id c637dbbb344dcc6186cb00dfa61cf4f4078bb8ebf8993b48d5114eed57c41698 Oct 30 06:27:33.195250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.195223 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6"] Oct 30 06:27:33.197138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.197104 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:33.197244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.197201 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:33.197331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.197250 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:33.437317 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.437281 5001 generic.go:356] "Generic (PLEG): container finished" podID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerID="1be51816e7cd52820608b7fc1dc43ca669c6c749a5aedb89cf3d25dcdaf20018" exitCode=0 Oct 30 06:27:33.437523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.437361 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" event={"ID":"1dbe1898-1454-48fe-8e9f-c1bcf0600b44","Type":"ContainerDied","Data":"1be51816e7cd52820608b7fc1dc43ca669c6c749a5aedb89cf3d25dcdaf20018"} Oct 30 06:27:33.437850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.437833 5001 scope.go:122] "RemoveContainer" containerID="1be51816e7cd52820608b7fc1dc43ca669c6c749a5aedb89cf3d25dcdaf20018" Oct 30 06:27:33.439527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.439499 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" event={"ID":"c741b57d-5029-4bfe-ab30-f58e5e45eb2c","Type":"ContainerStarted","Data":"eeecf4d942dd2b7787bc3e1fe21c83a7e3a3f20161a54e67d4be68a2e6616657"} Oct 30 06:27:33.439659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.439557 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" event={"ID":"c741b57d-5029-4bfe-ab30-f58e5e45eb2c","Type":"ContainerStarted","Data":"c637dbbb344dcc6186cb00dfa61cf4f4078bb8ebf8993b48d5114eed57c41698"} Oct 30 06:27:33.439659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.439598 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:33.439794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.439684 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:33.439794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.439728 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:33.441980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.441918 5001 generic.go:356] "Generic (PLEG): container finished" podID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerID="e88bddbf3606f0ae1de8c3391244724846231af8060b886a17b2e2e54189ec41" exitCode=0 Oct 30 06:27:33.442086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.442000 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" event={"ID":"c7f2a909-e267-4a73-89a1-37170fa5bc73","Type":"ContainerDied","Data":"e88bddbf3606f0ae1de8c3391244724846231af8060b886a17b2e2e54189ec41"} Oct 30 06:27:33.447406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.442297 5001 scope.go:122] "RemoveContainer" containerID="e88bddbf3606f0ae1de8c3391244724846231af8060b886a17b2e2e54189ec41" Oct 30 06:27:33.447406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.443553 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:33.447406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.443609 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:33.447406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.443647 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:33.592959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.592901 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" podStartSLOduration=1.592881594 podStartE2EDuration="1.592881594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:33.587584501 +0000 UTC m=+2067.583541512" watchObservedRunningTime="2026-10-30 06:27:33.592881594 +0000 UTC m=+2067.588838605" Oct 30 06:27:33.860809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:33.858460 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:34.462745 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:34.462709 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" event={"ID":"c7f2a909-e267-4a73-89a1-37170fa5bc73","Type":"ContainerStarted","Data":"7ffb1aa6bebd50764ae4cd7b5356eebb96c9adbe904eb907973c5e18eb6a60b1"} Oct 30 06:27:34.476403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:34.474245 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" event={"ID":"1dbe1898-1454-48fe-8e9f-c1bcf0600b44","Type":"ContainerStarted","Data":"8060131ed1d32b399e0275e31c63605774df70eff2fdb267cab81976a8a06e95"} Oct 30 06:27:34.550631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:34.550594 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:27:34.550852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:34.550769 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 06:27:34.550852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:34.550811 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 06:27:34.578337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:34.577958 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:27:34.593337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:34.585183 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 06:27:34.593337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:34.585246 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 06:27:34.885642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:34.885611 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:35.581802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:35.581763 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 06:27:35.621013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:35.620970 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 06:27:35.621627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:35.587740 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 06:27:35.621982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:35.621883 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 06:27:35.627669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:35.625989 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:35.628222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:35.628201 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:27:35.815411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:35.815371 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs"] Oct 30 06:27:35.815922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:35.815870 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" containerID="cri-o://de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064" gracePeriod=30 Oct 30 06:27:36.551061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:36.551025 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 06:27:36.551561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:36.551085 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 06:27:36.576330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:36.575928 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 06:27:36.576330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:36.575989 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 06:27:37.556518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:37.556484 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 06:27:37.557195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:37.557161 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 06:27:37.584330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:37.568878 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 06:27:37.584330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:37.568953 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 06:27:38.407332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:38.405593 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 06:27:38.555329 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:27:38.550397 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:27:38.555329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:38.550713 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 06:27:38.555329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:38.550768 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 06:27:38.731439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:38.708070 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:27:38.742717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:38.734401 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 06:27:38.742717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:38.734631 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 06:27:39.555334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:39.551737 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 06:27:39.555334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:39.551818 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 06:27:39.576468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:39.575254 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 06:27:39.576468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:39.575347 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 06:27:40.567459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:40.565475 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 30 06:27:40.567459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:40.565558 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 30 06:27:40.572340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:40.570407 5001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 06:27:40.572340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:40.570466 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 06:27:41.278331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.270508 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.337211 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-podinfo\") pod \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.337281 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-socket\") pod \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.337339 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-envoy\") pod \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.337408 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-credential-socket\") pod \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.337569 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-kube-api-access-w4xm5\" (UniqueName: \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-kube-api-access-w4xm5\") pod \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.337648 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-certs\") pod \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.337689 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-data\") pod \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.337729 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istiod-ca-cert\") pod \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.337865 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-token\" (UniqueName: \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-token\") pod \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\" (UID: \"6d6a40bf-862a-49e7-9fd5-a75582747c0d\") " Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.337962 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-credential-socket" pod "6d6a40bf-862a-49e7-9fd5-a75582747c0d" (UID: "6d6a40bf-862a-49e7-9fd5-a75582747c0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.338414 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istiod-ca-cert" pod "6d6a40bf-862a-49e7-9fd5-a75582747c0d" (UID: "6d6a40bf-862a-49e7-9fd5-a75582747c0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.338858 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-data" pod "6d6a40bf-862a-49e7-9fd5-a75582747c0d" (UID: "6d6a40bf-862a-49e7-9fd5-a75582747c0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.339072 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.339094 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.339110 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.339170 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-certs" pod "6d6a40bf-862a-49e7-9fd5-a75582747c0d" (UID: "6d6a40bf-862a-49e7-9fd5-a75582747c0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:41.341424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.339221 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-socket" pod "6d6a40bf-862a-49e7-9fd5-a75582747c0d" (UID: "6d6a40bf-862a-49e7-9fd5-a75582747c0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:41.342345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.341844 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-kube-api-access-w4xm5" pod "6d6a40bf-862a-49e7-9fd5-a75582747c0d" (UID: "6d6a40bf-862a-49e7-9fd5-a75582747c0d"). InnerVolumeSpecName "kube-api-access-w4xm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:41.342672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.342644 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-token" pod "6d6a40bf-862a-49e7-9fd5-a75582747c0d" (UID: "6d6a40bf-862a-49e7-9fd5-a75582747c0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:41.342784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.342684 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-envoy" pod "6d6a40bf-862a-49e7-9fd5-a75582747c0d" (UID: "6d6a40bf-862a-49e7-9fd5-a75582747c0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:41.347782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.347744 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-podinfo" pod "6d6a40bf-862a-49e7-9fd5-a75582747c0d" (UID: "6d6a40bf-862a-49e7-9fd5-a75582747c0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:27:41.441138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.441100 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:41.441138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.441140 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:41.441446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.441154 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:41.441446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.441169 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4xm5\" (UniqueName: \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-kube-api-access-w4xm5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:41.441446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.441185 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6a40bf-862a-49e7-9fd5-a75582747c0d-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:41.441446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.441198 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6a40bf-862a-49e7-9fd5-a75582747c0d-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:27:41.560556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.558599 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:27:41.560556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.560330 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:27:41.563774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.563726 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:27:41.588840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.588811 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:27:41.592412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.592384 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:27:41.811181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.811100 5001 generic.go:356] "Generic (PLEG): container finished" podID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerID="de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064" exitCode=0 Oct 30 06:27:41.811527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.811382 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" event={"ID":"6d6a40bf-862a-49e7-9fd5-a75582747c0d","Type":"ContainerDied","Data":"de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064"} Oct 30 06:27:41.811647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.811556 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" event={"ID":"6d6a40bf-862a-49e7-9fd5-a75582747c0d","Type":"ContainerDied","Data":"994b70315786465c001e9f34941ebd6c9de2cda820c4e63e7b7d0e0177fc03c9"} Oct 30 06:27:41.811647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.811407 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs" Oct 30 06:27:41.811757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.811585 5001 scope.go:122] "RemoveContainer" containerID="de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064" Oct 30 06:27:41.824662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.824518 5001 scope.go:122] "RemoveContainer" containerID="999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a" Oct 30 06:27:41.851055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.851032 5001 scope.go:122] "RemoveContainer" containerID="de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064" Oct 30 06:27:41.862337 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:27:41.861348 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064\": container with ID starting with de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064 not found: ID does not exist" containerID="de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064" Oct 30 06:27:41.862337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.861430 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064"} err="failed to get container status \"de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064\": rpc error: code = NotFound desc = could not find container \"de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064\": container with ID starting with de2e3a03de309bc1bfce53a04cfe65639d55f0e37c26befac9be9ef926c18064 not found: ID does not exist" Oct 30 06:27:41.862337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.861466 5001 scope.go:122] "RemoveContainer" containerID="999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a" Oct 30 06:27:41.862337 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:27:41.862079 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a\": container with ID starting with 999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a not found: ID does not exist" containerID="999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a" Oct 30 06:27:41.862337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.862116 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a"} err="failed to get container status \"999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a\": rpc error: code = NotFound desc = could not find container \"999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a\": container with ID starting with 999979c6c3864f27efb33d48db44ed311fcf0652d3a3fbee00cc20f14146840a not found: ID does not exist" Oct 30 06:27:41.885691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.885657 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs"] Oct 30 06:27:41.890612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:41.890586 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec27ed-openshift-default-676b7d8b4f-mghzs"] Oct 30 06:27:42.123671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:42.123631 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" path="/var/lib/kubelet/pods/6d6a40bf-862a-49e7-9fd5-a75582747c0d/volumes" Oct 30 06:27:45.993332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:45.992487 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9"] Oct 30 06:27:46.001341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:45.998589 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" Oct 30 06:27:46.001341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:45.998613 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" Oct 30 06:27:46.001341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:45.998634 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" Oct 30 06:27:46.001341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:45.998642 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" Oct 30 06:27:46.001341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:45.998867 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" Oct 30 06:27:46.001341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:45.998886 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6a40bf-862a-49e7-9fd5-a75582747c0d" containerName="istio-proxy" Oct 30 06:27:46.005382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.005361 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.012922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.012895 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9"] Oct 30 06:27:46.030497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.030450 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-3bbfc8-openshift-default-dockercfg-dhkhn\"" Oct 30 06:27:46.136340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.124703 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-certs\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.136689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.136655 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7vk\" (UniqueName: \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-kube-api-access-nd7vk\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.136887 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.136869 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-istio-token\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.155329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.137100 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-credential-socket\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.155651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.155623 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18bb796-a33f-41bc-93f2-302c700fd101-istio-podinfo\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.155778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.155761 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18bb796-a33f-41bc-93f2-302c700fd101-istiod-ca-cert\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.155893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.155878 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-socket\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.156084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.156065 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-data\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.156232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.156217 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-envoy\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.258193 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7vk\" (UniqueName: \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-kube-api-access-nd7vk\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.258288 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-istio-token\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.258491 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-credential-socket\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.258554 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18bb796-a33f-41bc-93f2-302c700fd101-istio-podinfo\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.258576 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18bb796-a33f-41bc-93f2-302c700fd101-istiod-ca-cert\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.258602 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-socket\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.258672 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-data\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.258715 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-envoy\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.258805 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-certs\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.267888 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-socket\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.268609 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18bb796-a33f-41bc-93f2-302c700fd101-istiod-ca-cert\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.269288 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-data\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.271378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.269842 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-credential-socket\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.272269 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.272244 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-certs\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.296377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.276616 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-envoy\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.333336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.331373 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7vk\" (UniqueName: \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-kube-api-access-nd7vk\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.334403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.334377 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18bb796-a33f-41bc-93f2-302c700fd101-istio-podinfo\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.350344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.345035 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-istio-token\") pod \"e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:46.636773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:46.636721 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:47.038020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:47.037994 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9"] Oct 30 06:27:47.038804 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:27:47.038782 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-dba2d59d3b397cf2e52df92f5009258b8bf8444e61f4ae7641de9109568d7380 WatchSource:0}: Error finding container dba2d59d3b397cf2e52df92f5009258b8bf8444e61f4ae7641de9109568d7380: Status 404 returned error can't find the container with id dba2d59d3b397cf2e52df92f5009258b8bf8444e61f4ae7641de9109568d7380 Oct 30 06:27:47.041491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:47.041458 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:47.041574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:47.041524 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:47.041574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:47.041552 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:47.909591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:47.909552 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" event={"ID":"d18bb796-a33f-41bc-93f2-302c700fd101","Type":"ContainerStarted","Data":"2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b"} Oct 30 06:27:47.909780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:47.909620 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" event={"ID":"d18bb796-a33f-41bc-93f2-302c700fd101","Type":"ContainerStarted","Data":"dba2d59d3b397cf2e52df92f5009258b8bf8444e61f4ae7641de9109568d7380"} Oct 30 06:27:47.981775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:47.981714 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" podStartSLOduration=2.981693254 podStartE2EDuration="2.981693254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:47.96480343 +0000 UTC m=+2081.960760441" watchObservedRunningTime="2026-10-30 06:27:47.981693254 +0000 UTC m=+2081.977650264" Oct 30 06:27:48.638436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:48.638397 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:48.645801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:48.645768 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:48.920196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:48.920114 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:48.921209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:48.921185 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:27:51.418702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:51.417206 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": read tcp 10.132.0.2:41954->10.132.1.13:15021: read: connection reset by peer" start-of-body= Oct 30 06:27:51.418702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:51.417289 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": read tcp 10.132.0.2:41954->10.132.1.13:15021: read: connection reset by peer" Oct 30 06:27:51.947966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:51.947935 5001 generic.go:356] "Generic (PLEG): container finished" podID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerID="7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817" exitCode=0 Oct 30 06:27:51.948170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:51.947987 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" event={"ID":"2fc22a5c-aff0-46dc-be59-b438bffb07a9","Type":"ContainerDied","Data":"7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817"} Oct 30 06:27:51.948501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:51.948480 5001 scope.go:122] "RemoveContainer" containerID="7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817" Oct 30 06:27:51.950585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:51.950548 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:51.950669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:51.950647 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:51.950711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:51.950693 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:52.544891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.544853 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm"] Oct 30 06:27:52.549941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.549912 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:27:52.571275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.569827 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-3bbfc8\"" Oct 30 06:27:52.571275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.569957 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-3bbfc8-dockercfg-5m5vf\"" Oct 30 06:27:52.579318 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.579257 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm"] Oct 30 06:27:52.617962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.617927 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c4bf28-4b50-4299-bd95-6b6e5124f684-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm\" (UID: \"39c4bf28-4b50-4299-bd95-6b6e5124f684\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:27:52.618160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.618039 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76sq\" (UniqueName: \"kubernetes.io/projected/39c4bf28-4b50-4299-bd95-6b6e5124f684-kube-api-access-c76sq\") pod \"payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm\" (UID: \"39c4bf28-4b50-4299-bd95-6b6e5124f684\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:27:52.641710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.641679 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2"] Oct 30 06:27:52.646599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.646574 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:27:52.732336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.726756 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76sq\" (UniqueName: \"kubernetes.io/projected/39c4bf28-4b50-4299-bd95-6b6e5124f684-kube-api-access-c76sq\") pod \"payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm\" (UID: \"39c4bf28-4b50-4299-bd95-6b6e5124f684\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:27:52.732336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.727057 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c4bf28-4b50-4299-bd95-6b6e5124f684-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm\" (UID: \"39c4bf28-4b50-4299-bd95-6b6e5124f684\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:27:52.732336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.727109 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5t5x\" (UniqueName: \"kubernetes.io/projected/d97f25ac-b719-4b21-986d-2afe9d0b39d6-kube-api-access-r5t5x\") pod \"payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2\" (UID: \"d97f25ac-b719-4b21-986d-2afe9d0b39d6\") " pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:27:52.732336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.727170 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97f25ac-b719-4b21-986d-2afe9d0b39d6-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2\" (UID: \"d97f25ac-b719-4b21-986d-2afe9d0b39d6\") " pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:27:52.732336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.729244 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c4bf28-4b50-4299-bd95-6b6e5124f684-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm\" (UID: \"39c4bf28-4b50-4299-bd95-6b6e5124f684\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:27:52.865541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.855368 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5t5x\" (UniqueName: \"kubernetes.io/projected/d97f25ac-b719-4b21-986d-2afe9d0b39d6-kube-api-access-r5t5x\") pod \"payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2\" (UID: \"d97f25ac-b719-4b21-986d-2afe9d0b39d6\") " pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:27:52.865541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:52.855437 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97f25ac-b719-4b21-986d-2afe9d0b39d6-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2\" (UID: \"d97f25ac-b719-4b21-986d-2afe9d0b39d6\") " pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:27:53.017577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.015662 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76sq\" (UniqueName: \"kubernetes.io/projected/39c4bf28-4b50-4299-bd95-6b6e5124f684-kube-api-access-c76sq\") pod \"payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm\" (UID: \"39c4bf28-4b50-4299-bd95-6b6e5124f684\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:27:53.017577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.016338 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97f25ac-b719-4b21-986d-2afe9d0b39d6-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2\" (UID: \"d97f25ac-b719-4b21-986d-2afe9d0b39d6\") " pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:27:53.024552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.024524 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:27:53.049119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.049077 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" event={"ID":"2fc22a5c-aff0-46dc-be59-b438bffb07a9","Type":"ContainerStarted","Data":"341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6"} Oct 30 06:27:53.087377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.081992 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2"] Oct 30 06:27:53.087377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.083018 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5t5x\" (UniqueName: \"kubernetes.io/projected/d97f25ac-b719-4b21-986d-2afe9d0b39d6-kube-api-access-r5t5x\") pod \"payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2\" (UID: \"d97f25ac-b719-4b21-986d-2afe9d0b39d6\") " pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:27:53.348898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.348861 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:27:53.402419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.402327 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm"] Oct 30 06:27:53.410564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.410544 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:27:53.411450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.411363 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 06:27:53.411450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.411407 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 06:27:53.715529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:53.715504 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2"] Oct 30 06:27:53.717115 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:27:53.717080 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97f25ac_b719_4b21_986d_2afe9d0b39d6.slice/crio-a2d93ab33fad08e4c982ddf56994bc419839075e6ffb4986e7fcf9f44dbc0c9d WatchSource:0}: Error finding container a2d93ab33fad08e4c982ddf56994bc419839075e6ffb4986e7fcf9f44dbc0c9d: Status 404 returned error can't find the container with id a2d93ab33fad08e4c982ddf56994bc419839075e6ffb4986e7fcf9f44dbc0c9d Oct 30 06:27:54.088529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.088359 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" event={"ID":"39c4bf28-4b50-4299-bd95-6b6e5124f684","Type":"ContainerStarted","Data":"8a406b03f9b32bb3ec66029266eb2698e1e4a4cc7064b5231e77314743fbd088"} Oct 30 06:27:54.088529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.088421 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" event={"ID":"39c4bf28-4b50-4299-bd95-6b6e5124f684","Type":"ContainerStarted","Data":"56b8a0a2b7e3462d9d65cb03d36a19df404556cd991d183feb8d94d244201a59"} Oct 30 06:27:54.088529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.088447 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:27:54.090185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.090156 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" event={"ID":"d97f25ac-b719-4b21-986d-2afe9d0b39d6","Type":"ContainerStarted","Data":"45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa"} Oct 30 06:27:54.090320 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.090200 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" event={"ID":"d97f25ac-b719-4b21-986d-2afe9d0b39d6","Type":"ContainerStarted","Data":"a2d93ab33fad08e4c982ddf56994bc419839075e6ffb4986e7fcf9f44dbc0c9d"} Oct 30 06:27:54.090320 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.090232 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:27:54.130263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.130216 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" podStartSLOduration=2.130201664 podStartE2EDuration="2.130201664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:54.127388984 +0000 UTC m=+2088.123345993" watchObservedRunningTime="2026-10-30 06:27:54.130201664 +0000 UTC m=+2088.126158712" Oct 30 06:27:54.186118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.186071 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" podStartSLOduration=2.186055176 podStartE2EDuration="2.186055176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:54.183138004 +0000 UTC m=+2088.179095018" watchObservedRunningTime="2026-10-30 06:27:54.186055176 +0000 UTC m=+2088.182012198" Oct 30 06:27:54.411732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.411688 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 06:27:54.411966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.411744 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 06:27:54.412404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.412375 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9"] Oct 30 06:27:54.417491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.417468 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:27:54.434022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.433950 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-b9ff93-dockercfg-pkpxr\"" Oct 30 06:27:54.434191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.434117 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-b9ff93\"" Oct 30 06:27:54.444630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.444607 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9"] Oct 30 06:27:54.489823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.489783 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw6g4\" (UniqueName: \"kubernetes.io/projected/fc4ef9dd-2519-4506-a03e-3205ad65d331-kube-api-access-qw6g4\") pod \"payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9\" (UID: \"fc4ef9dd-2519-4506-a03e-3205ad65d331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:27:54.491498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.491469 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4ef9dd-2519-4506-a03e-3205ad65d331-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9\" (UID: \"fc4ef9dd-2519-4506-a03e-3205ad65d331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:27:54.504348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.499219 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh"] Oct 30 06:27:54.593341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.590561 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh"] Oct 30 06:27:54.593341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.590717 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:27:54.629331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.617367 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw6g4\" (UniqueName: \"kubernetes.io/projected/fc4ef9dd-2519-4506-a03e-3205ad65d331-kube-api-access-qw6g4\") pod \"payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9\" (UID: \"fc4ef9dd-2519-4506-a03e-3205ad65d331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:27:54.629331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.617489 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4ef9dd-2519-4506-a03e-3205ad65d331-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9\" (UID: \"fc4ef9dd-2519-4506-a03e-3205ad65d331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:27:54.629331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.618460 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4ef9dd-2519-4506-a03e-3205ad65d331-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9\" (UID: \"fc4ef9dd-2519-4506-a03e-3205ad65d331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:27:54.842132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.830442 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw6g4\" (UniqueName: \"kubernetes.io/projected/fc4ef9dd-2519-4506-a03e-3205ad65d331-kube-api-access-qw6g4\") pod \"payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9\" (UID: \"fc4ef9dd-2519-4506-a03e-3205ad65d331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:27:54.842132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.841709 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635a07fa-78b2-457e-9f7e-f41d6610b469-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh\" (UID: \"635a07fa-78b2-457e-9f7e-f41d6610b469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:27:54.842132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.841839 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ssdt\" (UniqueName: \"kubernetes.io/projected/635a07fa-78b2-457e-9f7e-f41d6610b469-kube-api-access-9ssdt\") pod \"payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh\" (UID: \"635a07fa-78b2-457e-9f7e-f41d6610b469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:27:54.846132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.843948 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:27:54.944256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.944209 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635a07fa-78b2-457e-9f7e-f41d6610b469-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh\" (UID: \"635a07fa-78b2-457e-9f7e-f41d6610b469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:27:54.944415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.944364 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ssdt\" (UniqueName: \"kubernetes.io/projected/635a07fa-78b2-457e-9f7e-f41d6610b469-kube-api-access-9ssdt\") pod \"payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh\" (UID: \"635a07fa-78b2-457e-9f7e-f41d6610b469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:27:54.945793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.945767 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635a07fa-78b2-457e-9f7e-f41d6610b469-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh\" (UID: \"635a07fa-78b2-457e-9f7e-f41d6610b469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:27:54.994995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:54.993352 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ssdt\" (UniqueName: \"kubernetes.io/projected/635a07fa-78b2-457e-9f7e-f41d6610b469-kube-api-access-9ssdt\") pod \"payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh\" (UID: \"635a07fa-78b2-457e-9f7e-f41d6610b469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:27:55.190327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:55.190284 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:27:55.222980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:55.222950 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9"] Oct 30 06:27:55.223260 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:27:55.223175 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4ef9dd_2519_4506_a03e_3205ad65d331.slice/crio-1490ea8960fd06883682878a48272061a7dee650ddd4921f5fa4cbf5af275f49 WatchSource:0}: Error finding container 1490ea8960fd06883682878a48272061a7dee650ddd4921f5fa4cbf5af275f49: Status 404 returned error can't find the container with id 1490ea8960fd06883682878a48272061a7dee650ddd4921f5fa4cbf5af275f49 Oct 30 06:27:55.410874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:55.410844 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 06:27:55.411046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:55.410899 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 06:27:55.543321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:55.543272 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh"] Oct 30 06:27:55.543866 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:27:55.543840 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635a07fa_78b2_457e_9f7e_f41d6610b469.slice/crio-55904650a4f0064e57b82d8ce91fb278997e824c291a0b82cd8a7c64cee387ae WatchSource:0}: Error finding container 55904650a4f0064e57b82d8ce91fb278997e824c291a0b82cd8a7c64cee387ae: Status 404 returned error can't find the container with id 55904650a4f0064e57b82d8ce91fb278997e824c291a0b82cd8a7c64cee387ae Oct 30 06:27:56.124267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:56.124225 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" event={"ID":"fc4ef9dd-2519-4506-a03e-3205ad65d331","Type":"ContainerStarted","Data":"ad9506eaf4662347ba1dcff4533f3198fe58863b1649c9dba63b44dc754acc29"} Oct 30 06:27:56.124703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:56.124284 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" event={"ID":"fc4ef9dd-2519-4506-a03e-3205ad65d331","Type":"ContainerStarted","Data":"1490ea8960fd06883682878a48272061a7dee650ddd4921f5fa4cbf5af275f49"} Oct 30 06:27:56.124703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:56.124330 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:27:56.124703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:56.124352 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" event={"ID":"635a07fa-78b2-457e-9f7e-f41d6610b469","Type":"ContainerStarted","Data":"fe7cb610461fd97d6aaf39540832119ab0fe1db5acf1996bda71b54acfc85dc0"} Oct 30 06:27:56.124703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:56.124380 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" event={"ID":"635a07fa-78b2-457e-9f7e-f41d6610b469","Type":"ContainerStarted","Data":"55904650a4f0064e57b82d8ce91fb278997e824c291a0b82cd8a7c64cee387ae"} Oct 30 06:27:56.124703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:56.124408 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:27:56.173327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:56.171665 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" podStartSLOduration=2.171641343 podStartE2EDuration="2.171641343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:56.157706449 +0000 UTC m=+2090.153663461" watchObservedRunningTime="2026-10-30 06:27:56.171641343 +0000 UTC m=+2090.167598355" Oct 30 06:27:56.203235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:56.203187 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" podStartSLOduration=2.203174677 podStartE2EDuration="2.203174677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:56.202241798 +0000 UTC m=+2090.198198807" watchObservedRunningTime="2026-10-30 06:27:56.203174677 +0000 UTC m=+2090.199131687" Oct 30 06:27:56.411076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:56.410990 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 06:27:56.411076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:56.411050 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 06:27:57.411099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:57.411064 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 06:27:57.411603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:57.411134 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 06:27:58.513525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:58.513486 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:27:58.514159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:58.514129 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:27:58.515766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:27:58.515743 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:28:05.105922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:05.105886 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:28:05.106524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:05.105982 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:28:05.257729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:05.257322 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:28:06.719406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:06.719369 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:28:06.732982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:06.732941 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:28:07.133447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:07.133418 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:28:07.133777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:07.133756 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:28:08.479911 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:28:08.479832 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:28:09.033724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:09.033690 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97"] Oct 30 06:28:09.101761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:09.101722 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2fdd6fc-mnc97"] Oct 30 06:28:10.120549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:10.120516 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3450d685-8949-4d2f-a820-bbc371682fc2" path="/var/lib/kubelet/pods/3450d685-8949-4d2f-a820-bbc371682fc2/volumes" Oct 30 06:28:17.545511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:17.545479 5001 generic.go:356] "Generic (PLEG): container finished" podID="4549b201-684a-49f1-b871-4c0866e27bbb" containerID="f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7" exitCode=0 Oct 30 06:28:17.545992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:17.545554 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" event={"ID":"4549b201-684a-49f1-b871-4c0866e27bbb","Type":"ContainerDied","Data":"f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7"} Oct 30 06:28:17.546044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:17.545991 5001 scope.go:122] "RemoveContainer" containerID="f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7" Oct 30 06:28:17.547775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:17.547741 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:17.547935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:17.547825 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:17.547935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:17.547875 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:18.668329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:18.653874 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" event={"ID":"4549b201-684a-49f1-b871-4c0866e27bbb","Type":"ContainerStarted","Data":"83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d"} Oct 30 06:28:19.561331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:19.559677 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:28:19.561331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:19.560407 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 30 06:28:19.561331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:19.560462 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 30 06:28:20.524636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:20.524603 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:28:20.561693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:20.561656 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 30 06:28:20.561997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:20.561969 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 30 06:28:21.560573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:21.560534 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 30 06:28:21.561203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:21.561171 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 30 06:28:22.574750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:22.574713 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 30 06:28:22.575399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:22.575365 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 30 06:28:23.562332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:23.560633 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 30 06:28:23.562332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:23.560694 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 30 06:28:24.574329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:24.573345 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:28:24.582578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:24.582547 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 06:28:33.886094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:33.886009 5001 generic.go:356] "Generic (PLEG): container finished" podID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerID="eeecf4d942dd2b7787bc3e1fe21c83a7e3a3f20161a54e67d4be68a2e6616657" exitCode=0 Oct 30 06:28:33.886094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:33.886080 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" event={"ID":"c741b57d-5029-4bfe-ab30-f58e5e45eb2c","Type":"ContainerDied","Data":"eeecf4d942dd2b7787bc3e1fe21c83a7e3a3f20161a54e67d4be68a2e6616657"} Oct 30 06:28:33.886670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:33.886650 5001 scope.go:122] "RemoveContainer" containerID="eeecf4d942dd2b7787bc3e1fe21c83a7e3a3f20161a54e67d4be68a2e6616657" Oct 30 06:28:33.888831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:33.888792 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:33.888956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:33.888890 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:33.888956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:33.888938 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:34.914346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:34.914283 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" event={"ID":"c741b57d-5029-4bfe-ab30-f58e5e45eb2c","Type":"ContainerStarted","Data":"e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c"} Oct 30 06:28:35.626036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:35.625999 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:28:35.859279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:35.859243 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:28:35.859691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:35.859663 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 06:28:35.859873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:35.859848 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 06:28:36.865329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:36.864736 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 06:28:36.865329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:36.864820 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 06:28:37.858769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:37.858730 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 06:28:37.859649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:37.859614 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 06:28:38.294580 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:28:38.294483 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:28:38.859330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:38.857730 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 06:28:38.859330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:38.857809 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 06:28:39.878430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:39.878390 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 06:28:39.879057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:39.879020 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 06:28:40.863911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:40.863873 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:28:40.864960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:40.864935 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 06:28:48.200023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:48.199983 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:28:59.046582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:59.046548 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb"] Oct 30 06:28:59.057487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:28:59.057450 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb9d77b-sb4mb"] Oct 30 06:29:00.119117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:00.119083 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81fb235e-78b9-4f68-8698-e62153540562" path="/var/lib/kubelet/pods/81fb235e-78b9-4f68-8698-e62153540562/volumes" Oct 30 06:29:00.297500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:00.297468 5001 generic.go:356] "Generic (PLEG): container finished" podID="d18bb796-a33f-41bc-93f2-302c700fd101" containerID="2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b" exitCode=0 Oct 30 06:29:00.297709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:00.297505 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" event={"ID":"d18bb796-a33f-41bc-93f2-302c700fd101","Type":"ContainerDied","Data":"2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b"} Oct 30 06:29:00.297964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:00.297948 5001 scope.go:122] "RemoveContainer" containerID="2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b" Oct 30 06:29:00.299788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:00.299754 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:29:00.299923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:00.299848 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:29:00.299923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:00.299903 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:29:01.311878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:01.311839 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" event={"ID":"d18bb796-a33f-41bc-93f2-302c700fd101","Type":"ContainerStarted","Data":"686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927"} Oct 30 06:29:01.638113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:01.638078 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:29:01.639105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:01.639082 5001 patch_prober.go:29] interesting pod/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Oct 30 06:29:01.639227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:01.639145 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Oct 30 06:29:02.638261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:02.638226 5001 patch_prober.go:29] interesting pod/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Oct 30 06:29:02.638698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:02.638279 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Oct 30 06:29:03.637939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:03.637903 5001 patch_prober.go:29] interesting pod/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Oct 30 06:29:03.638128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:03.637960 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Oct 30 06:29:03.920578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:03.920496 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:29:04.637779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:04.637744 5001 patch_prober.go:29] interesting pod/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Oct 30 06:29:04.638112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:04.637803 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Oct 30 06:29:05.638268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:05.638228 5001 patch_prober.go:29] interesting pod/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Oct 30 06:29:05.638791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:05.638287 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Oct 30 06:29:06.638637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:06.638601 5001 patch_prober.go:29] interesting pod/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body= Oct 30 06:29:06.639046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:06.638652 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" Oct 30 06:29:07.820488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:07.820456 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:29:07.833631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:07.833600 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:29:08.272190 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:29:08.272144 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:29:09.194102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:09.194067 5001 scope.go:122] "RemoveContainer" containerID="31a37f228bbba2d2ecc997e24aec2ae8a22db595a445a316e89f8d965ec84efd" Oct 30 06:29:09.208487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:09.208466 5001 scope.go:122] "RemoveContainer" containerID="63b3d982306c19da3a7f17053185605c8adeca00c633e27506beaa739c88b254" Oct 30 06:29:09.218900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:09.218872 5001 scope.go:122] "RemoveContainer" containerID="97bc3645e28f72d3811025d766214737bd7a418ffea5cc61206731545f866ac3" Oct 30 06:29:09.228990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:09.228966 5001 scope.go:122] "RemoveContainer" containerID="796ba6eb405daa951a903d74176acc18e535dd07276e41c88f94ad78b565ef09" Oct 30 06:29:11.034145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:11.034105 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8"] Oct 30 06:29:11.045236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:11.045199 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398718f0-jvzq8"] Oct 30 06:29:12.119108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:12.119071 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc1bfd1-6c91-484b-8caa-768fe57d18b9" path="/var/lib/kubelet/pods/afc1bfd1-6c91-484b-8caa-768fe57d18b9/volumes" Oct 30 06:29:14.035097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:14.035058 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5"] Oct 30 06:29:14.045961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:14.045933 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb3d825-6kzk5"] Oct 30 06:29:14.118872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:14.118839 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d24bf2c-e5ce-40ea-b861-8aa56e57603f" path="/var/lib/kubelet/pods/0d24bf2c-e5ce-40ea-b861-8aa56e57603f/volumes" Oct 30 06:29:24.033402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:24.033365 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4"] Oct 30 06:29:24.044469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:24.044444 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb32039-s55w4"] Oct 30 06:29:24.119119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:24.119088 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e274757d-d67e-4857-8dee-db81b94c65a7" path="/var/lib/kubelet/pods/e274757d-d67e-4857-8dee-db81b94c65a7/volumes" Oct 30 06:29:25.489335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:25.489265 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:29:27.037035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:27.036994 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7"] Oct 30 06:29:27.075376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:27.062281 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38b19e07-kzxl7"] Oct 30 06:29:27.983572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:27.983538 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt"] Oct 30 06:29:27.987230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:27.987211 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:28.006921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.006888 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt"] Oct 30 06:29:28.011064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.011037 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:29:28.011266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.011093 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:29:28.011969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.011946 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:29:28.013741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.013717 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:29:28.045953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.045918 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:28.046419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.045960 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9rj7\" (UniqueName: \"kubernetes.io/projected/ebe11027-c91f-42bf-91f9-35c15a23da51-kube-api-access-p9rj7\") pod \"maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:28.046419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.046186 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:28.119950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.119915 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15722fe8-c60d-4e9d-b5c2-bbe3b7b49947" path="/var/lib/kubelet/pods/15722fe8-c60d-4e9d-b5c2-bbe3b7b49947/volumes" Oct 30 06:29:28.148115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.148078 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:28.148296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.148222 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:28.148296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.148258 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9rj7\" (UniqueName: \"kubernetes.io/projected/ebe11027-c91f-42bf-91f9-35c15a23da51-kube-api-access-p9rj7\") pod \"maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:28.148988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.148950 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:28.150679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.150653 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:28.171316 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.171277 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9rj7\" (UniqueName: \"kubernetes.io/projected/ebe11027-c91f-42bf-91f9-35c15a23da51-kube-api-access-p9rj7\") pod \"maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:28.305818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.305729 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:28.626658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.626634 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt"] Oct 30 06:29:28.630707 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:29:28.630668 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe11027_c91f_42bf_91f9_35c15a23da51.slice/crio-23a728e40362523cb643ccded4b33ebe5917e7244e8acdb1a35271ec461e693e WatchSource:0}: Error finding container 23a728e40362523cb643ccded4b33ebe5917e7244e8acdb1a35271ec461e693e: Status 404 returned error can't find the container with id 23a728e40362523cb643ccded4b33ebe5917e7244e8acdb1a35271ec461e693e Oct 30 06:29:28.721673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:28.721636 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" event={"ID":"ebe11027-c91f-42bf-91f9-35c15a23da51","Type":"ContainerStarted","Data":"23a728e40362523cb643ccded4b33ebe5917e7244e8acdb1a35271ec461e693e"} Oct 30 06:29:29.732578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:29.732542 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" event={"ID":"ebe11027-c91f-42bf-91f9-35c15a23da51","Type":"ContainerStarted","Data":"bc871d1d032940840b4a81d1c0cc19d4aca47da0364065d25dd65a832642eca4"} Oct 30 06:29:29.776692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:29.776638 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" podStartSLOduration=2.776623251 podStartE2EDuration="2.776623251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:29:29.77075219 +0000 UTC m=+2183.766709200" watchObservedRunningTime="2026-10-30 06:29:29.776623251 +0000 UTC m=+2183.772580297" Oct 30 06:29:30.743685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:30.743640 5001 generic.go:356] "Generic (PLEG): container finished" podID="ebe11027-c91f-42bf-91f9-35c15a23da51" containerID="bc871d1d032940840b4a81d1c0cc19d4aca47da0364065d25dd65a832642eca4" exitCode=0 Oct 30 06:29:30.744154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:30.743728 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" event={"ID":"ebe11027-c91f-42bf-91f9-35c15a23da51","Type":"ContainerDied","Data":"bc871d1d032940840b4a81d1c0cc19d4aca47da0364065d25dd65a832642eca4"} Oct 30 06:29:31.936422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:31.936394 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:32.008889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.008800 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-openshift-ca\") pod \"ebe11027-c91f-42bf-91f9-35c15a23da51\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " Oct 30 06:29:32.008889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.008856 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-xks-ca\") pod \"ebe11027-c91f-42bf-91f9-35c15a23da51\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " Oct 30 06:29:32.009180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.009159 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe11027-c91f-42bf-91f9-35c15a23da51-kube-api-access-p9rj7\" (UniqueName: \"kubernetes.io/projected/ebe11027-c91f-42bf-91f9-35c15a23da51-kube-api-access-p9rj7\") pod \"ebe11027-c91f-42bf-91f9-35c15a23da51\" (UID: \"ebe11027-c91f-42bf-91f9-35c15a23da51\") " Oct 30 06:29:32.009258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.009235 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-openshift-ca" pod "ebe11027-c91f-42bf-91f9-35c15a23da51" (UID: "ebe11027-c91f-42bf-91f9-35c15a23da51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:29:32.009848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.009804 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:29:32.011343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.011317 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-xks-ca" pod "ebe11027-c91f-42bf-91f9-35c15a23da51" (UID: "ebe11027-c91f-42bf-91f9-35c15a23da51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:29:32.012011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.011984 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe11027-c91f-42bf-91f9-35c15a23da51-kube-api-access-p9rj7" pod "ebe11027-c91f-42bf-91f9-35c15a23da51" (UID: "ebe11027-c91f-42bf-91f9-35c15a23da51"). InnerVolumeSpecName "kube-api-access-p9rj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:29:32.111494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.111462 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9rj7\" (UniqueName: \"kubernetes.io/projected/ebe11027-c91f-42bf-91f9-35c15a23da51-kube-api-access-p9rj7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:29:32.111675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.111491 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe11027-c91f-42bf-91f9-35c15a23da51-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:29:32.765844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.765812 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" event={"ID":"ebe11027-c91f-42bf-91f9-35c15a23da51","Type":"ContainerDied","Data":"23a728e40362523cb643ccded4b33ebe5917e7244e8acdb1a35271ec461e693e"} Oct 30 06:29:32.766025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.765866 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23a728e40362523cb643ccded4b33ebe5917e7244e8acdb1a35271ec461e693e" Oct 30 06:29:32.766025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:32.765818 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt" Oct 30 06:29:35.037011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:35.036977 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh"] Oct 30 06:29:35.048409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:35.048375 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c61850-bd7sh"] Oct 30 06:29:36.118727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:36.118696 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675dcd63-75cb-451b-999d-79b1ac5017b9" path="/var/lib/kubelet/pods/675dcd63-75cb-451b-999d-79b1ac5017b9/volumes" Oct 30 06:29:38.267876 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:29:38.267835 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:29:38.431360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:38.431321 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2"] Oct 30 06:29:38.464355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:38.464324 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm"] Oct 30 06:29:42.033797 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:42.033756 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml"] Oct 30 06:29:42.047166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:42.047133 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b59102a7-8mxml"] Oct 30 06:29:42.138182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:42.138149 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e" path="/var/lib/kubelet/pods/ca0f0bd6-41ce-4056-8f64-fd9f1c75e91e/volumes" Oct 30 06:29:43.431828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.431785 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" podUID="d97f25ac-b719-4b21-986d-2afe9d0b39d6" containerName="payload-processing" containerID="cri-o://45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa" gracePeriod=25 Oct 30 06:29:43.464982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.464950 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" podUID="39c4bf28-4b50-4299-bd95-6b6e5124f684" containerName="payload-pre-processing" containerID="cri-o://8a406b03f9b32bb3ec66029266eb2698e1e4a4cc7064b5231e77314743fbd088" gracePeriod=25 Oct 30 06:29:43.792377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.789803 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:29:43.898432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.898399 5001 generic.go:356] "Generic (PLEG): container finished" podID="d97f25ac-b719-4b21-986d-2afe9d0b39d6" containerID="45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa" exitCode=0 Oct 30 06:29:43.898568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.898452 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" Oct 30 06:29:43.898925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.898674 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" event={"ID":"d97f25ac-b719-4b21-986d-2afe9d0b39d6","Type":"ContainerDied","Data":"45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa"} Oct 30 06:29:43.898925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.898746 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2" event={"ID":"d97f25ac-b719-4b21-986d-2afe9d0b39d6","Type":"ContainerDied","Data":"a2d93ab33fad08e4c982ddf56994bc419839075e6ffb4986e7fcf9f44dbc0c9d"} Oct 30 06:29:43.898925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.898778 5001 scope.go:122] "RemoveContainer" containerID="45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa" Oct 30 06:29:43.900900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.900878 5001 generic.go:356] "Generic (PLEG): container finished" podID="39c4bf28-4b50-4299-bd95-6b6e5124f684" containerID="8a406b03f9b32bb3ec66029266eb2698e1e4a4cc7064b5231e77314743fbd088" exitCode=0 Oct 30 06:29:43.901006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.900936 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" event={"ID":"39c4bf28-4b50-4299-bd95-6b6e5124f684","Type":"ContainerDied","Data":"8a406b03f9b32bb3ec66029266eb2698e1e4a4cc7064b5231e77314743fbd088"} Oct 30 06:29:43.913672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.913649 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:29:43.915812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.915793 5001 scope.go:122] "RemoveContainer" containerID="45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa" Oct 30 06:29:43.916117 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:29:43.916096 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa\": container with ID starting with 45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa not found: ID does not exist" containerID="45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa" Oct 30 06:29:43.916198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.916125 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa"} err="failed to get container status \"45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa\": rpc error: code = NotFound desc = could not find container \"45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa\": container with ID starting with 45decf006ac5214ef1a0ba334d7cf8f05d68e3fc9cfc7b55011611f1966cadaa not found: ID does not exist" Oct 30 06:29:43.931537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.931510 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d97f25ac-b719-4b21-986d-2afe9d0b39d6-kube-api-access-r5t5x\" (UniqueName: \"kubernetes.io/projected/d97f25ac-b719-4b21-986d-2afe9d0b39d6-kube-api-access-r5t5x\") pod \"d97f25ac-b719-4b21-986d-2afe9d0b39d6\" (UID: \"d97f25ac-b719-4b21-986d-2afe9d0b39d6\") " Oct 30 06:29:43.931683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.931577 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d97f25ac-b719-4b21-986d-2afe9d0b39d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97f25ac-b719-4b21-986d-2afe9d0b39d6-plugins-config-volume\") pod \"d97f25ac-b719-4b21-986d-2afe9d0b39d6\" (UID: \"d97f25ac-b719-4b21-986d-2afe9d0b39d6\") " Oct 30 06:29:43.932136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.932106 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97f25ac-b719-4b21-986d-2afe9d0b39d6-plugins-config-volume" pod "d97f25ac-b719-4b21-986d-2afe9d0b39d6" (UID: "d97f25ac-b719-4b21-986d-2afe9d0b39d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:29:43.932281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.932264 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97f25ac-b719-4b21-986d-2afe9d0b39d6-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:29:43.934230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:43.934209 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97f25ac-b719-4b21-986d-2afe9d0b39d6-kube-api-access-r5t5x" pod "d97f25ac-b719-4b21-986d-2afe9d0b39d6" (UID: "d97f25ac-b719-4b21-986d-2afe9d0b39d6"). InnerVolumeSpecName "kube-api-access-r5t5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:29:44.033572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.033492 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c4bf28-4b50-4299-bd95-6b6e5124f684-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c4bf28-4b50-4299-bd95-6b6e5124f684-plugins-config-volume\") pod \"39c4bf28-4b50-4299-bd95-6b6e5124f684\" (UID: \"39c4bf28-4b50-4299-bd95-6b6e5124f684\") " Oct 30 06:29:44.033572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.033545 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c4bf28-4b50-4299-bd95-6b6e5124f684-kube-api-access-c76sq\" (UniqueName: \"kubernetes.io/projected/39c4bf28-4b50-4299-bd95-6b6e5124f684-kube-api-access-c76sq\") pod \"39c4bf28-4b50-4299-bd95-6b6e5124f684\" (UID: \"39c4bf28-4b50-4299-bd95-6b6e5124f684\") " Oct 30 06:29:44.033884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.033856 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c4bf28-4b50-4299-bd95-6b6e5124f684-plugins-config-volume" pod "39c4bf28-4b50-4299-bd95-6b6e5124f684" (UID: "39c4bf28-4b50-4299-bd95-6b6e5124f684"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:29:44.034374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.034355 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5t5x\" (UniqueName: \"kubernetes.io/projected/d97f25ac-b719-4b21-986d-2afe9d0b39d6-kube-api-access-r5t5x\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:29:44.034374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.034376 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39c4bf28-4b50-4299-bd95-6b6e5124f684-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:29:44.035938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.035917 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c4bf28-4b50-4299-bd95-6b6e5124f684-kube-api-access-c76sq" pod "39c4bf28-4b50-4299-bd95-6b6e5124f684" (UID: "39c4bf28-4b50-4299-bd95-6b6e5124f684"). InnerVolumeSpecName "kube-api-access-c76sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:29:44.135605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.135576 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c76sq\" (UniqueName: \"kubernetes.io/projected/39c4bf28-4b50-4299-bd95-6b6e5124f684-kube-api-access-c76sq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:29:44.240122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.240086 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2"] Oct 30 06:29:44.244880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.244858 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3bbfc8-7fdcdbf498-7zzq2"] Oct 30 06:29:44.914072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.914034 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" Oct 30 06:29:44.914845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.914094 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm" event={"ID":"39c4bf28-4b50-4299-bd95-6b6e5124f684","Type":"ContainerDied","Data":"56b8a0a2b7e3462d9d65cb03d36a19df404556cd991d183feb8d94d244201a59"} Oct 30 06:29:44.914845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.914154 5001 scope.go:122] "RemoveContainer" containerID="8a406b03f9b32bb3ec66029266eb2698e1e4a4cc7064b5231e77314743fbd088" Oct 30 06:29:44.970987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.970947 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm"] Oct 30 06:29:44.979552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:44.979522 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3bbfc8-6554f4994c-rnmrm"] Oct 30 06:29:46.118854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:46.118825 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c4bf28-4b50-4299-bd95-6b6e5124f684" path="/var/lib/kubelet/pods/39c4bf28-4b50-4299-bd95-6b6e5124f684/volumes" Oct 30 06:29:46.119270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:46.119256 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d97f25ac-b719-4b21-986d-2afe9d0b39d6" path="/var/lib/kubelet/pods/d97f25ac-b719-4b21-986d-2afe9d0b39d6/volumes" Oct 30 06:29:51.036174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:51.036090 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p"] Oct 30 06:29:51.047182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:51.047158 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7a71b7-rjx7p"] Oct 30 06:29:52.118866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:52.118833 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb586cb3-d7bf-4ad7-9249-8555759a7bd0" path="/var/lib/kubelet/pods/eb586cb3-d7bf-4ad7-9249-8555759a7bd0/volumes" Oct 30 06:29:53.684197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.684156 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht"] Oct 30 06:29:53.685244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.685217 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d97f25ac-b719-4b21-986d-2afe9d0b39d6" containerName="payload-processing" Oct 30 06:29:53.685244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.685245 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d97f25ac-b719-4b21-986d-2afe9d0b39d6" containerName="payload-processing" Oct 30 06:29:53.685463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.685295 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ebe11027-c91f-42bf-91f9-35c15a23da51" containerName="revoke-keys" Oct 30 06:29:53.685463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.685320 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe11027-c91f-42bf-91f9-35c15a23da51" containerName="revoke-keys" Oct 30 06:29:53.685463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.685351 5001 cpu_manager.go:414] "containerMap: removing container" podUID="39c4bf28-4b50-4299-bd95-6b6e5124f684" containerName="payload-pre-processing" Oct 30 06:29:53.685463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.685358 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c4bf28-4b50-4299-bd95-6b6e5124f684" containerName="payload-pre-processing" Oct 30 06:29:53.685701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.685582 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe11027-c91f-42bf-91f9-35c15a23da51" containerName="revoke-keys" Oct 30 06:29:53.685701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.685601 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c4bf28-4b50-4299-bd95-6b6e5124f684" containerName="payload-pre-processing" Oct 30 06:29:53.685701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.685617 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d97f25ac-b719-4b21-986d-2afe9d0b39d6" containerName="payload-processing" Oct 30 06:29:53.692140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.692115 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.706755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.706723 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht"] Oct 30 06:29:53.717423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.717377 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-b9ff93\"/\"kube-root-ca.crt\"" Oct 30 06:29:53.717617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.717577 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-b9ff93\"/\"auth-test-model-b9ff93-kserve-self-signed-certs\"" Oct 30 06:29:53.721359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.721211 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-b9ff93\"/\"openshift-service-ca.crt\"" Oct 30 06:29:53.721359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.721215 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-b9ff93\"/\"default-dockercfg-ffps8\"" Oct 30 06:29:53.779719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.779683 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-model-cache\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.779931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.779902 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a96c90-0ab3-4482-91b9-cb9282434c1c-tls-certs\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.779982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.779962 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmml\" (UniqueName: \"kubernetes.io/projected/84a96c90-0ab3-4482-91b9-cb9282434c1c-kube-api-access-klmml\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.780219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.780111 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-home\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.780219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.780148 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-dshm\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.780544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.780518 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-tmp-dir\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.882355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.882243 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a96c90-0ab3-4482-91b9-cb9282434c1c-tls-certs\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.882355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.882355 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmml\" (UniqueName: \"kubernetes.io/projected/84a96c90-0ab3-4482-91b9-cb9282434c1c-kube-api-access-klmml\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.882628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.882517 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-home\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.882628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.882556 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-dshm\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.882628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.882608 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-tmp-dir\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.882808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.882678 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-model-cache\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.883277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.883250 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-model-cache\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.883440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.883418 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-home\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.883925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.883894 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-tmp-dir\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.886056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.886029 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-dshm\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.886536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.886516 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a96c90-0ab3-4482-91b9-cb9282434c1c-tls-certs\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:53.906046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:53.906019 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmml\" (UniqueName: \"kubernetes.io/projected/84a96c90-0ab3-4482-91b9-cb9282434c1c-kube-api-access-klmml\") pod \"auth-test-model-b9ff93-kserve-7fb4f48569-p8zht\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:54.029516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:54.029428 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:54.423836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:54.423809 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht"] Oct 30 06:29:54.427044 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:29:54.426993 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84a96c90_0ab3_4482_91b9_cb9282434c1c.slice/crio-6dc28bea7ec14e1064e796874b192cd8887cd8c65e671730d04b55cfaf310244 WatchSource:0}: Error finding container 6dc28bea7ec14e1064e796874b192cd8887cd8c65e671730d04b55cfaf310244: Status 404 returned error can't find the container with id 6dc28bea7ec14e1064e796874b192cd8887cd8c65e671730d04b55cfaf310244 Oct 30 06:29:55.067373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:55.067206 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj"] Oct 30 06:29:55.072419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:55.072365 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" event={"ID":"84a96c90-0ab3-4482-91b9-cb9282434c1c","Type":"ContainerStarted","Data":"8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972"} Oct 30 06:29:55.072680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:55.072656 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" event={"ID":"84a96c90-0ab3-4482-91b9-cb9282434c1c","Type":"ContainerStarted","Data":"6dc28bea7ec14e1064e796874b192cd8887cd8c65e671730d04b55cfaf310244"} Oct 30 06:29:55.133538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:55.131077 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d43f67bf-bhjtj"] Oct 30 06:29:56.040122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:56.040080 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:29:56.120809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:29:56.120767 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c" path="/var/lib/kubelet/pods/73ebe4b0-8e32-4173-b3ea-141ab9e3fd8c/volumes" Oct 30 06:30:00.146579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.146512 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" podStartSLOduration=7.146473706 podStartE2EDuration="7.146473706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:29:55.131443403 +0000 UTC m=+2209.127400415" watchObservedRunningTime="2026-10-30 06:30:00.146473706 +0000 UTC m=+2214.142430716" Oct 30 06:30:00.148793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.148683 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx"] Oct 30 06:30:00.153975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.153947 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" Oct 30 06:30:00.154427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.154403 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq"] Oct 30 06:30:00.158941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.158915 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" Oct 30 06:30:00.159582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.159565 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq"] Oct 30 06:30:00.165000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.164974 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" Oct 30 06:30:00.165982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.165963 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6"] Oct 30 06:30:00.170252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.170232 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" Oct 30 06:30:00.171595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.171574 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr"] Oct 30 06:30:00.172692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.172672 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:30:00.173082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.172776 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:30:00.173426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.173405 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:30:00.181864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.181837 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" Oct 30 06:30:00.182349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.182290 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5"] Oct 30 06:30:00.207011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.206986 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" Oct 30 06:30:00.234906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.232843 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx"] Oct 30 06:30:00.240678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.240651 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj"] Oct 30 06:30:00.241631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.241605 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbkq\" (UniqueName: \"kubernetes.io/projected/2a0a42e5-b9fc-4eff-b9da-4969663d6027-kube-api-access-9pbkq\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq\" (UID: \"2a0a42e5-b9fc-4eff-b9da-4969663d6027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" Oct 30 06:30:00.241734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.241656 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc67p\" (UniqueName: \"kubernetes.io/projected/4ac79889-b8fc-40c1-a988-946e1525dd2a-kube-api-access-zc67p\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx\" (UID: \"4ac79889-b8fc-40c1-a988-946e1525dd2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" Oct 30 06:30:00.241778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.241765 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5\" (UID: \"6a0ea84e-5daf-494f-b5e6-be8757eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" Oct 30 06:30:00.241825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.241810 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr\" (UID: \"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" Oct 30 06:30:00.241860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.241841 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcbd5\" (UniqueName: \"kubernetes.io/projected/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-kube-api-access-qcbd5\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5\" (UID: \"6a0ea84e-5daf-494f-b5e6-be8757eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" Oct 30 06:30:00.241909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.241896 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0a42e5-b9fc-4eff-b9da-4969663d6027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq\" (UID: \"2a0a42e5-b9fc-4eff-b9da-4969663d6027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" Oct 30 06:30:00.241946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.241933 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f91032-e116-4d28-bc15-a4b5144777da-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889030-f8mcq\" (UID: \"f6f91032-e116-4d28-bc15-a4b5144777da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" Oct 30 06:30:00.241982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.241958 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6\" (UID: \"1f0799a8-e504-4570-ba51-1b2aadcbfd1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" Oct 30 06:30:00.242317 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.242279 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac79889-b8fc-40c1-a988-946e1525dd2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx\" (UID: \"4ac79889-b8fc-40c1-a988-946e1525dd2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" Oct 30 06:30:00.242437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.242412 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krzgz\" (UniqueName: \"kubernetes.io/projected/f6f91032-e116-4d28-bc15-a4b5144777da-kube-api-access-krzgz\") pod \"maas-api-key-cleanup-29889030-f8mcq\" (UID: \"f6f91032-e116-4d28-bc15-a4b5144777da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" Oct 30 06:30:00.242526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.242510 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fblqk\" (UniqueName: \"kubernetes.io/projected/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-kube-api-access-fblqk\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6\" (UID: \"1f0799a8-e504-4570-ba51-1b2aadcbfd1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" Oct 30 06:30:00.242584 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.242571 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2tz\" (UniqueName: \"kubernetes.io/projected/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-kube-api-access-tk2tz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr\" (UID: \"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" Oct 30 06:30:00.256243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.255094 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq"] Oct 30 06:30:00.256243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.255212 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" Oct 30 06:30:00.256243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.255988 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4"] Oct 30 06:30:00.265089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.265066 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq"] Oct 30 06:30:00.265292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.265180 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" Oct 30 06:30:00.272368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.272345 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr"] Oct 30 06:30:00.277329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.277291 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6"] Oct 30 06:30:00.284918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.284895 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq"] Oct 30 06:30:00.290965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.290945 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" Oct 30 06:30:00.292400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.292379 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5"] Oct 30 06:30:00.297609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.297587 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv"] Oct 30 06:30:00.304742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.303217 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" Oct 30 06:30:00.306903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.306881 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj"] Oct 30 06:30:00.311840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.311818 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq"] Oct 30 06:30:00.316156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.316135 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv"] Oct 30 06:30:00.322212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.322186 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4"] Oct 30 06:30:00.344014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.343982 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjr2\" (UniqueName: \"kubernetes.io/projected/4f638843-af28-44fe-a19a-b2a04a84db0e-kube-api-access-2jjr2\") pod \"maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv\" (UID: \"4f638843-af28-44fe-a19a-b2a04a84db0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" Oct 30 06:30:00.344212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344047 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fblqk\" (UniqueName: \"kubernetes.io/projected/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-kube-api-access-fblqk\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6\" (UID: \"1f0799a8-e504-4570-ba51-1b2aadcbfd1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" Oct 30 06:30:00.344212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344194 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2tz\" (UniqueName: \"kubernetes.io/projected/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-kube-api-access-tk2tz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr\" (UID: \"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" Oct 30 06:30:00.344371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344237 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5393b2-ac60-4342-8185-8f77a0212db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4\" (UID: \"3f5393b2-ac60-4342-8185-8f77a0212db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" Oct 30 06:30:00.344371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344285 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj\" (UID: \"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" Oct 30 06:30:00.344371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344344 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gq6\" (UniqueName: \"kubernetes.io/projected/78c12b75-839e-427e-8cb2-9d3873ea8ee4-kube-api-access-q5gq6\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq\" (UID: \"78c12b75-839e-427e-8cb2-9d3873ea8ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" Oct 30 06:30:00.344539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344404 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbkq\" (UniqueName: \"kubernetes.io/projected/2a0a42e5-b9fc-4eff-b9da-4969663d6027-kube-api-access-9pbkq\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq\" (UID: \"2a0a42e5-b9fc-4eff-b9da-4969663d6027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" Oct 30 06:30:00.344539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344442 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc67p\" (UniqueName: \"kubernetes.io/projected/4ac79889-b8fc-40c1-a988-946e1525dd2a-kube-api-access-zc67p\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx\" (UID: \"4ac79889-b8fc-40c1-a988-946e1525dd2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" Oct 30 06:30:00.344745 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344722 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5\" (UID: \"6a0ea84e-5daf-494f-b5e6-be8757eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" Oct 30 06:30:00.344854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344768 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c12b75-839e-427e-8cb2-9d3873ea8ee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq\" (UID: \"78c12b75-839e-427e-8cb2-9d3873ea8ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" Oct 30 06:30:00.344854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344791 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr\" (UID: \"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" Oct 30 06:30:00.344854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344810 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcbd5\" (UniqueName: \"kubernetes.io/projected/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-kube-api-access-qcbd5\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5\" (UID: \"6a0ea84e-5daf-494f-b5e6-be8757eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" Oct 30 06:30:00.345024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344927 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0a42e5-b9fc-4eff-b9da-4969663d6027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq\" (UID: \"2a0a42e5-b9fc-4eff-b9da-4969663d6027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" Oct 30 06:30:00.345024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.344976 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq899\" (UniqueName: \"kubernetes.io/projected/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-kube-api-access-jq899\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj\" (UID: \"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" Oct 30 06:30:00.345712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.345693 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0a42e5-b9fc-4eff-b9da-4969663d6027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq\" (UID: \"2a0a42e5-b9fc-4eff-b9da-4969663d6027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" Oct 30 06:30:00.345783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.345611 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr\" (UID: \"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" Oct 30 06:30:00.345783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.345707 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f91032-e116-4d28-bc15-a4b5144777da-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889030-f8mcq\" (UID: \"f6f91032-e116-4d28-bc15-a4b5144777da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" Oct 30 06:30:00.345783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.345389 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5\" (UID: \"6a0ea84e-5daf-494f-b5e6-be8757eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" Oct 30 06:30:00.345783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.345752 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6\" (UID: \"1f0799a8-e504-4570-ba51-1b2aadcbfd1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" Oct 30 06:30:00.345944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.345924 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jhzr\" (UniqueName: \"kubernetes.io/projected/3f5393b2-ac60-4342-8185-8f77a0212db0-kube-api-access-9jhzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4\" (UID: \"3f5393b2-ac60-4342-8185-8f77a0212db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" Oct 30 06:30:00.346034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.346015 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac79889-b8fc-40c1-a988-946e1525dd2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx\" (UID: \"4ac79889-b8fc-40c1-a988-946e1525dd2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" Oct 30 06:30:00.346097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.346055 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f638843-af28-44fe-a19a-b2a04a84db0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv\" (UID: \"4f638843-af28-44fe-a19a-b2a04a84db0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" Oct 30 06:30:00.346097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.346090 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krzgz\" (UniqueName: \"kubernetes.io/projected/f6f91032-e116-4d28-bc15-a4b5144777da-kube-api-access-krzgz\") pod \"maas-api-key-cleanup-29889030-f8mcq\" (UID: \"f6f91032-e116-4d28-bc15-a4b5144777da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" Oct 30 06:30:00.346506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.346481 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f91032-e116-4d28-bc15-a4b5144777da-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889030-f8mcq\" (UID: \"f6f91032-e116-4d28-bc15-a4b5144777da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" Oct 30 06:30:00.346612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.346510 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6\" (UID: \"1f0799a8-e504-4570-ba51-1b2aadcbfd1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" Oct 30 06:30:00.346692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.346650 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac79889-b8fc-40c1-a988-946e1525dd2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx\" (UID: \"4ac79889-b8fc-40c1-a988-946e1525dd2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" Oct 30 06:30:00.368710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.368621 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc67p\" (UniqueName: \"kubernetes.io/projected/4ac79889-b8fc-40c1-a988-946e1525dd2a-kube-api-access-zc67p\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx\" (UID: \"4ac79889-b8fc-40c1-a988-946e1525dd2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" Oct 30 06:30:00.368710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.368669 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fblqk\" (UniqueName: \"kubernetes.io/projected/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-kube-api-access-fblqk\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6\" (UID: \"1f0799a8-e504-4570-ba51-1b2aadcbfd1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" Oct 30 06:30:00.368710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.368625 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2tz\" (UniqueName: \"kubernetes.io/projected/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-kube-api-access-tk2tz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr\" (UID: \"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" Oct 30 06:30:00.368948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.368855 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krzgz\" (UniqueName: \"kubernetes.io/projected/f6f91032-e116-4d28-bc15-a4b5144777da-kube-api-access-krzgz\") pod \"maas-api-key-cleanup-29889030-f8mcq\" (UID: \"f6f91032-e116-4d28-bc15-a4b5144777da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" Oct 30 06:30:00.368948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.368855 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcbd5\" (UniqueName: \"kubernetes.io/projected/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-kube-api-access-qcbd5\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5\" (UID: \"6a0ea84e-5daf-494f-b5e6-be8757eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" Oct 30 06:30:00.369368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.369347 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbkq\" (UniqueName: \"kubernetes.io/projected/2a0a42e5-b9fc-4eff-b9da-4969663d6027-kube-api-access-9pbkq\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq\" (UID: \"2a0a42e5-b9fc-4eff-b9da-4969663d6027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" Oct 30 06:30:00.447950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.447843 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c12b75-839e-427e-8cb2-9d3873ea8ee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq\" (UID: \"78c12b75-839e-427e-8cb2-9d3873ea8ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" Oct 30 06:30:00.448274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.448246 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq899\" (UniqueName: \"kubernetes.io/projected/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-kube-api-access-jq899\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj\" (UID: \"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" Oct 30 06:30:00.448483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.448455 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jhzr\" (UniqueName: \"kubernetes.io/projected/3f5393b2-ac60-4342-8185-8f77a0212db0-kube-api-access-9jhzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4\" (UID: \"3f5393b2-ac60-4342-8185-8f77a0212db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" Oct 30 06:30:00.448634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.448525 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f638843-af28-44fe-a19a-b2a04a84db0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv\" (UID: \"4f638843-af28-44fe-a19a-b2a04a84db0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" Oct 30 06:30:00.448634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.448605 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjr2\" (UniqueName: \"kubernetes.io/projected/4f638843-af28-44fe-a19a-b2a04a84db0e-kube-api-access-2jjr2\") pod \"maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv\" (UID: \"4f638843-af28-44fe-a19a-b2a04a84db0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" Oct 30 06:30:00.448757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.448704 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5393b2-ac60-4342-8185-8f77a0212db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4\" (UID: \"3f5393b2-ac60-4342-8185-8f77a0212db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" Oct 30 06:30:00.448836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.448760 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj\" (UID: \"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" Oct 30 06:30:00.448836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.448800 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gq6\" (UniqueName: \"kubernetes.io/projected/78c12b75-839e-427e-8cb2-9d3873ea8ee4-kube-api-access-q5gq6\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq\" (UID: \"78c12b75-839e-427e-8cb2-9d3873ea8ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" Oct 30 06:30:00.449831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.449799 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c12b75-839e-427e-8cb2-9d3873ea8ee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq\" (UID: \"78c12b75-839e-427e-8cb2-9d3873ea8ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" Oct 30 06:30:00.450350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.449985 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj\" (UID: \"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" Oct 30 06:30:00.450350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.450233 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f638843-af28-44fe-a19a-b2a04a84db0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv\" (UID: \"4f638843-af28-44fe-a19a-b2a04a84db0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" Oct 30 06:30:00.450793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.450767 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5393b2-ac60-4342-8185-8f77a0212db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4\" (UID: \"3f5393b2-ac60-4342-8185-8f77a0212db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" Oct 30 06:30:00.471873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.471825 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jhzr\" (UniqueName: \"kubernetes.io/projected/3f5393b2-ac60-4342-8185-8f77a0212db0-kube-api-access-9jhzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4\" (UID: \"3f5393b2-ac60-4342-8185-8f77a0212db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" Oct 30 06:30:00.472083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.471825 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq899\" (UniqueName: \"kubernetes.io/projected/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-kube-api-access-jq899\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj\" (UID: \"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" Oct 30 06:30:00.472083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.471917 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gq6\" (UniqueName: \"kubernetes.io/projected/78c12b75-839e-427e-8cb2-9d3873ea8ee4-kube-api-access-q5gq6\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq\" (UID: \"78c12b75-839e-427e-8cb2-9d3873ea8ee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" Oct 30 06:30:00.472217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.472157 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjr2\" (UniqueName: \"kubernetes.io/projected/4f638843-af28-44fe-a19a-b2a04a84db0e-kube-api-access-2jjr2\") pod \"maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv\" (UID: \"4f638843-af28-44fe-a19a-b2a04a84db0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" Oct 30 06:30:00.472917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.472889 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" Oct 30 06:30:00.498786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.498746 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" Oct 30 06:30:00.528194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.528152 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" Oct 30 06:30:00.533600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.533578 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" Oct 30 06:30:00.560748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.560496 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" Oct 30 06:30:00.579460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.577544 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" Oct 30 06:30:00.590937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.590916 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" Oct 30 06:30:00.601490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.597712 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" Oct 30 06:30:00.621466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.621433 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" Oct 30 06:30:00.629993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.629929 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" Oct 30 06:30:00.821857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:00.821829 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx"] Oct 30 06:30:01.034253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.034220 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz"] Oct 30 06:30:01.039071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.039046 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8f26ca3-wkhjz"] Oct 30 06:30:01.049155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.049126 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq"] Oct 30 06:30:01.050731 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:01.050695 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice/crio-dd0771f76d511ee6079e45f22950dad76c1e161c4d8dba17007dd902e914372d WatchSource:0}: Error finding container dd0771f76d511ee6079e45f22950dad76c1e161c4d8dba17007dd902e914372d: Status 404 returned error can't find the container with id dd0771f76d511ee6079e45f22950dad76c1e161c4d8dba17007dd902e914372d Oct 30 06:30:01.102814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.102777 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6"] Oct 30 06:30:01.109147 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:01.109112 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0799a8_e504_4570_ba51_1b2aadcbfd1f.slice/crio-4c95af7c4c12621a6bc61e326bc2fdaf0bc0292cb5a55e0f584a59fe9163024f WatchSource:0}: Error finding container 4c95af7c4c12621a6bc61e326bc2fdaf0bc0292cb5a55e0f584a59fe9163024f: Status 404 returned error can't find the container with id 4c95af7c4c12621a6bc61e326bc2fdaf0bc0292cb5a55e0f584a59fe9163024f Oct 30 06:30:01.142539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.142505 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr"] Oct 30 06:30:01.154150 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:01.154106 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc73e_5fcd_4a34_9465_ddbd8a86ccd6.slice/crio-ecf8a6c5a7aef9da99d44a218d23353eb9480c22ae65c61ee7a2a50eef081b2f WatchSource:0}: Error finding container ecf8a6c5a7aef9da99d44a218d23353eb9480c22ae65c61ee7a2a50eef081b2f: Status 404 returned error can't find the container with id ecf8a6c5a7aef9da99d44a218d23353eb9480c22ae65c61ee7a2a50eef081b2f Oct 30 06:30:01.160120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.160079 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" event={"ID":"2a0a42e5-b9fc-4eff-b9da-4969663d6027","Type":"ContainerStarted","Data":"dd0771f76d511ee6079e45f22950dad76c1e161c4d8dba17007dd902e914372d"} Oct 30 06:30:01.166043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.166007 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" event={"ID":"4ac79889-b8fc-40c1-a988-946e1525dd2a","Type":"ContainerStarted","Data":"b6bc6c1e3c06310171e3a01ac1ca39876b09cc1e974474eb43b5a217e1988996"} Oct 30 06:30:01.168242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.168206 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" event={"ID":"1f0799a8-e504-4570-ba51-1b2aadcbfd1f","Type":"ContainerStarted","Data":"4c95af7c4c12621a6bc61e326bc2fdaf0bc0292cb5a55e0f584a59fe9163024f"} Oct 30 06:30:01.172897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.172869 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq"] Oct 30 06:30:01.230905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.230847 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5"] Oct 30 06:30:01.255108 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:01.255052 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f91032_e116_4d28_bc15_a4b5144777da.slice/crio-cc423f27728a1d6ed2bc6ccd5a2b5e3dbeb9c2b04e807743184a71700692e19b WatchSource:0}: Error finding container cc423f27728a1d6ed2bc6ccd5a2b5e3dbeb9c2b04e807743184a71700692e19b: Status 404 returned error can't find the container with id cc423f27728a1d6ed2bc6ccd5a2b5e3dbeb9c2b04e807743184a71700692e19b Oct 30 06:30:01.295131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.295103 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq"] Oct 30 06:30:01.299729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.299700 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj"] Oct 30 06:30:01.306762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.306734 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4"] Oct 30 06:30:01.338596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:01.338566 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv"] Oct 30 06:30:01.357496 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:01.356188 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice/crio-599c3717587b32299096e201ee530bed7e96dc856958cae2bd29f9afb90aedd2 WatchSource:0}: Error finding container 599c3717587b32299096e201ee530bed7e96dc856958cae2bd29f9afb90aedd2: Status 404 returned error can't find the container with id 599c3717587b32299096e201ee530bed7e96dc856958cae2bd29f9afb90aedd2 Oct 30 06:30:01.362693 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:01.361539 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5393b2_ac60_4342_8185_8f77a0212db0.slice/crio-7963982d9a32b2c62f1198e076a401cc6cdbd50ddc859c0cb2e9a623a8864f3c WatchSource:0}: Error finding container 7963982d9a32b2c62f1198e076a401cc6cdbd50ddc859c0cb2e9a623a8864f3c: Status 404 returned error can't find the container with id 7963982d9a32b2c62f1198e076a401cc6cdbd50ddc859c0cb2e9a623a8864f3c Oct 30 06:30:01.368463 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:01.368441 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f638843_af28_44fe_a19a_b2a04a84db0e.slice/crio-86ade3abfe0e486bcc49636f43908e6c5f88ae9a4385380fc7250b9235981d4f WatchSource:0}: Error finding container 86ade3abfe0e486bcc49636f43908e6c5f88ae9a4385380fc7250b9235981d4f: Status 404 returned error can't find the container with id 86ade3abfe0e486bcc49636f43908e6c5f88ae9a4385380fc7250b9235981d4f Oct 30 06:30:02.121446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.121409 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="071940ca-738b-477f-a37b-6f179bc82f8b" path="/var/lib/kubelet/pods/071940ca-738b-477f-a37b-6f179bc82f8b/volumes" Oct 30 06:30:02.179156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.179069 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" event={"ID":"f6f91032-e116-4d28-bc15-a4b5144777da","Type":"ContainerStarted","Data":"cc423f27728a1d6ed2bc6ccd5a2b5e3dbeb9c2b04e807743184a71700692e19b"} Oct 30 06:30:02.180693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.180665 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" event={"ID":"3f5393b2-ac60-4342-8185-8f77a0212db0","Type":"ContainerStarted","Data":"7963982d9a32b2c62f1198e076a401cc6cdbd50ddc859c0cb2e9a623a8864f3c"} Oct 30 06:30:02.182643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.182613 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" event={"ID":"2a0a42e5-b9fc-4eff-b9da-4969663d6027","Type":"ContainerStarted","Data":"fd41ddfb8625fa5aa6d857327049d930e3c586cdd1d96729bfce987134989b14"} Oct 30 06:30:02.184493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.184466 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" event={"ID":"4ac79889-b8fc-40c1-a988-946e1525dd2a","Type":"ContainerStarted","Data":"3910a769a769d8d928062f8ce2a699480c3c75324ca7d91cdfb1dc581958673c"} Oct 30 06:30:02.185782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.185757 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" event={"ID":"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6","Type":"ContainerStarted","Data":"ecf8a6c5a7aef9da99d44a218d23353eb9480c22ae65c61ee7a2a50eef081b2f"} Oct 30 06:30:02.187343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.187314 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" event={"ID":"1f0799a8-e504-4570-ba51-1b2aadcbfd1f","Type":"ContainerStarted","Data":"5b518f8e4a7c0cc4f579c07810eba762fa72e757fe695364c126c26322709d67"} Oct 30 06:30:02.194483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.194451 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" event={"ID":"78c12b75-839e-427e-8cb2-9d3873ea8ee4","Type":"ContainerStarted","Data":"ab314e5ed407bd91d58882f479fdc643d61698b146bacc502f439e595e6a12fa"} Oct 30 06:30:02.196780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.196748 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" event={"ID":"4f638843-af28-44fe-a19a-b2a04a84db0e","Type":"ContainerStarted","Data":"86ade3abfe0e486bcc49636f43908e6c5f88ae9a4385380fc7250b9235981d4f"} Oct 30 06:30:02.202143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.202113 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" event={"ID":"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb","Type":"ContainerStarted","Data":"9aa61823ec349a71f7b5222b4723f81ae099e155de0410266565320a8502f448"} Oct 30 06:30:02.203597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.203576 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" event={"ID":"6a0ea84e-5daf-494f-b5e6-be8757eccb1b","Type":"ContainerStarted","Data":"599c3717587b32299096e201ee530bed7e96dc856958cae2bd29f9afb90aedd2"} Oct 30 06:30:02.225354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.225291 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" podStartSLOduration=2.225279115 podStartE2EDuration="2.225279115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:02.22027708 +0000 UTC m=+2216.216234090" watchObservedRunningTime="2026-10-30 06:30:02.225279115 +0000 UTC m=+2216.221236124" Oct 30 06:30:02.261577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.261098 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" podStartSLOduration=2.261077587 podStartE2EDuration="2.261077587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:02.258671556 +0000 UTC m=+2216.254628566" watchObservedRunningTime="2026-10-30 06:30:02.261077587 +0000 UTC m=+2216.257034598" Oct 30 06:30:02.302117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:02.302071 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" podStartSLOduration=2.302055828 podStartE2EDuration="2.302055828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:02.298088277 +0000 UTC m=+2216.294045288" watchObservedRunningTime="2026-10-30 06:30:02.302055828 +0000 UTC m=+2216.298012837" Oct 30 06:30:03.216426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.216395 5001 generic.go:356] "Generic (PLEG): container finished" podID="4f638843-af28-44fe-a19a-b2a04a84db0e" containerID="b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b" exitCode=60 Oct 30 06:30:03.216889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.216483 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" event={"ID":"4f638843-af28-44fe-a19a-b2a04a84db0e","Type":"ContainerDied","Data":"b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b"} Oct 30 06:30:03.216889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.216809 5001 scope.go:122] "RemoveContainer" containerID="b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b" Oct 30 06:30:03.218578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.218558 5001 generic.go:356] "Generic (PLEG): container finished" podID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerID="f2fb2a9e1191b07f3cc6520242c7761615dcab7714956cf063ff4323e68abccc" exitCode=60 Oct 30 06:30:03.218700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.218582 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" event={"ID":"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb","Type":"ContainerDied","Data":"f2fb2a9e1191b07f3cc6520242c7761615dcab7714956cf063ff4323e68abccc"} Oct 30 06:30:03.218939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.218919 5001 scope.go:122] "RemoveContainer" containerID="f2fb2a9e1191b07f3cc6520242c7761615dcab7714956cf063ff4323e68abccc" Oct 30 06:30:03.220535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.220437 5001 generic.go:356] "Generic (PLEG): container finished" podID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerID="793e0ad298e6ddaa6f7def81ba16750427daee80b0e076a0b6f97aa11c65c01e" exitCode=60 Oct 30 06:30:03.220535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.220459 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" event={"ID":"6a0ea84e-5daf-494f-b5e6-be8757eccb1b","Type":"ContainerDied","Data":"793e0ad298e6ddaa6f7def81ba16750427daee80b0e076a0b6f97aa11c65c01e"} Oct 30 06:30:03.220740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.220723 5001 scope.go:122] "RemoveContainer" containerID="793e0ad298e6ddaa6f7def81ba16750427daee80b0e076a0b6f97aa11c65c01e" Oct 30 06:30:03.222896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.222876 5001 generic.go:356] "Generic (PLEG): container finished" podID="f6f91032-e116-4d28-bc15-a4b5144777da" containerID="ec0ec08df473ecf884fd6fb4376cc37ce054f1ea6e06756a6c1da8b0d86927b6" exitCode=60 Oct 30 06:30:03.222998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.222976 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" event={"ID":"f6f91032-e116-4d28-bc15-a4b5144777da","Type":"ContainerDied","Data":"ec0ec08df473ecf884fd6fb4376cc37ce054f1ea6e06756a6c1da8b0d86927b6"} Oct 30 06:30:03.223204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.223149 5001 scope.go:122] "RemoveContainer" containerID="ec0ec08df473ecf884fd6fb4376cc37ce054f1ea6e06756a6c1da8b0d86927b6" Oct 30 06:30:03.224840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.224822 5001 generic.go:356] "Generic (PLEG): container finished" podID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerID="a1de637d0f7f03e14b581003e46f43c718ba3eadeee90f60ffbe514bfaa016a2" exitCode=60 Oct 30 06:30:03.224957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.224892 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" event={"ID":"3f5393b2-ac60-4342-8185-8f77a0212db0","Type":"ContainerDied","Data":"a1de637d0f7f03e14b581003e46f43c718ba3eadeee90f60ffbe514bfaa016a2"} Oct 30 06:30:03.225082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.225047 5001 scope.go:122] "RemoveContainer" containerID="a1de637d0f7f03e14b581003e46f43c718ba3eadeee90f60ffbe514bfaa016a2" Oct 30 06:30:03.226796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.226776 5001 generic.go:356] "Generic (PLEG): container finished" podID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerID="fd41ddfb8625fa5aa6d857327049d930e3c586cdd1d96729bfce987134989b14" exitCode=60 Oct 30 06:30:03.226913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.226861 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" event={"ID":"2a0a42e5-b9fc-4eff-b9da-4969663d6027","Type":"ContainerDied","Data":"fd41ddfb8625fa5aa6d857327049d930e3c586cdd1d96729bfce987134989b14"} Oct 30 06:30:03.227061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.227044 5001 scope.go:122] "RemoveContainer" containerID="fd41ddfb8625fa5aa6d857327049d930e3c586cdd1d96729bfce987134989b14" Oct 30 06:30:03.228947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.228930 5001 generic.go:356] "Generic (PLEG): container finished" podID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerID="3910a769a769d8d928062f8ce2a699480c3c75324ca7d91cdfb1dc581958673c" exitCode=60 Oct 30 06:30:03.229038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.228980 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" event={"ID":"4ac79889-b8fc-40c1-a988-946e1525dd2a","Type":"ContainerDied","Data":"3910a769a769d8d928062f8ce2a699480c3c75324ca7d91cdfb1dc581958673c"} Oct 30 06:30:03.229252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.229230 5001 scope.go:122] "RemoveContainer" containerID="3910a769a769d8d928062f8ce2a699480c3c75324ca7d91cdfb1dc581958673c" Oct 30 06:30:03.232005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.231716 5001 generic.go:356] "Generic (PLEG): container finished" podID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerID="febbf1684fa6988bb9fc9367649ce38f09c3ef2b939ad07d47e1d8bf33532d9f" exitCode=60 Oct 30 06:30:03.232005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.231833 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" event={"ID":"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6","Type":"ContainerDied","Data":"febbf1684fa6988bb9fc9367649ce38f09c3ef2b939ad07d47e1d8bf33532d9f"} Oct 30 06:30:03.232475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.232454 5001 scope.go:122] "RemoveContainer" containerID="febbf1684fa6988bb9fc9367649ce38f09c3ef2b939ad07d47e1d8bf33532d9f" Oct 30 06:30:03.234411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.234216 5001 generic.go:356] "Generic (PLEG): container finished" podID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerID="5b518f8e4a7c0cc4f579c07810eba762fa72e757fe695364c126c26322709d67" exitCode=60 Oct 30 06:30:03.234482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.234418 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" event={"ID":"1f0799a8-e504-4570-ba51-1b2aadcbfd1f","Type":"ContainerDied","Data":"5b518f8e4a7c0cc4f579c07810eba762fa72e757fe695364c126c26322709d67"} Oct 30 06:30:03.235222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.235137 5001 scope.go:122] "RemoveContainer" containerID="5b518f8e4a7c0cc4f579c07810eba762fa72e757fe695364c126c26322709d67" Oct 30 06:30:03.236875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.236854 5001 generic.go:356] "Generic (PLEG): container finished" podID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerID="91ff4a304ffa78284c60e0e84c5694db5b15b8a98314541cf44781b75c8a0f62" exitCode=60 Oct 30 06:30:03.236986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.236892 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" event={"ID":"78c12b75-839e-427e-8cb2-9d3873ea8ee4","Type":"ContainerDied","Data":"91ff4a304ffa78284c60e0e84c5694db5b15b8a98314541cf44781b75c8a0f62"} Oct 30 06:30:03.237724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.237372 5001 scope.go:122] "RemoveContainer" containerID="91ff4a304ffa78284c60e0e84c5694db5b15b8a98314541cf44781b75c8a0f62" Oct 30 06:30:03.352537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:03.352507 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv"] Oct 30 06:30:04.040646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:04.040599 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:30:04.056486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:04.056454 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:30:04.222879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:04.222538 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt"] Oct 30 06:30:04.222879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:04.222585 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3bbfc8-b4crt"] Oct 30 06:30:04.248778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:04.248740 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" event={"ID":"4f638843-af28-44fe-a19a-b2a04a84db0e","Type":"ContainerStarted","Data":"d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e"} Oct 30 06:30:04.248981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:04.248827 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" podUID="4f638843-af28-44fe-a19a-b2a04a84db0e" containerName="cleanup" containerID="cri-o://d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e" gracePeriod=30 Oct 30 06:30:04.252702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:04.252675 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" event={"ID":"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb","Type":"ContainerStarted","Data":"473c6f2b1016998acd2d3cc90be7eba6846f3e6e9f11fbede8717a4446f849ed"} Oct 30 06:30:04.266840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:04.266809 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:30:04.289942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:04.289884 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" podStartSLOduration=4.289866187 podStartE2EDuration="4.289866187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:04.286819047 +0000 UTC m=+2218.282776071" watchObservedRunningTime="2026-10-30 06:30:04.289866187 +0000 UTC m=+2218.285823200" Oct 30 06:30:04.325915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:04.325869 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" podStartSLOduration=4.325857057 podStartE2EDuration="4.325857057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:04.324963186 +0000 UTC m=+2218.320920197" watchObservedRunningTime="2026-10-30 06:30:04.325857057 +0000 UTC m=+2218.321814109" Oct 30 06:30:05.271788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.271753 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" event={"ID":"4ac79889-b8fc-40c1-a988-946e1525dd2a","Type":"ContainerStarted","Data":"b5cc5c401c5bfefa18ce44f74d3a32bde96f42c5172299b86559ee7427ce6dc6"} Oct 30 06:30:05.274081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.274053 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" event={"ID":"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6","Type":"ContainerStarted","Data":"10a94a1738b598a7f2add23a59a35871dff7b36b2c525e12427f1842c7461930"} Oct 30 06:30:05.276527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.276500 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" event={"ID":"1f0799a8-e504-4570-ba51-1b2aadcbfd1f","Type":"ContainerStarted","Data":"c1e6df6f71a347a9c96821cc3325d9098b00a2411662796b8b606a686b384bda"} Oct 30 06:30:05.276724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.276709 5001 scope.go:122] "RemoveContainer" containerID="c1e6df6f71a347a9c96821cc3325d9098b00a2411662796b8b606a686b384bda" Oct 30 06:30:05.277028 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:05.277007 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6_odh-ai-gateway-infra(1f0799a8-e504-4570-ba51-1b2aadcbfd1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" Oct 30 06:30:05.278844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.278814 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" event={"ID":"78c12b75-839e-427e-8cb2-9d3873ea8ee4","Type":"ContainerStarted","Data":"c30e6bc92024f442735467c8568c75dfcf2a4066e21b4ae8160c1ff197e8c9f9"} Oct 30 06:30:05.280888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.280867 5001 generic.go:356] "Generic (PLEG): container finished" podID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerID="473c6f2b1016998acd2d3cc90be7eba6846f3e6e9f11fbede8717a4446f849ed" exitCode=60 Oct 30 06:30:05.280980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.280947 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" event={"ID":"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb","Type":"ContainerDied","Data":"473c6f2b1016998acd2d3cc90be7eba6846f3e6e9f11fbede8717a4446f849ed"} Oct 30 06:30:05.281046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.280999 5001 scope.go:122] "RemoveContainer" containerID="f2fb2a9e1191b07f3cc6520242c7761615dcab7714956cf063ff4323e68abccc" Oct 30 06:30:05.281126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.281110 5001 scope.go:122] "RemoveContainer" containerID="473c6f2b1016998acd2d3cc90be7eba6846f3e6e9f11fbede8717a4446f849ed" Oct 30 06:30:05.281446 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:05.281418 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj_odh-ai-gateway-infra(a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" Oct 30 06:30:05.283253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.283230 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" event={"ID":"6a0ea84e-5daf-494f-b5e6-be8757eccb1b","Type":"ContainerStarted","Data":"403c5f8865d3c446f48e3d3635aed23bd2df68c750c17c1ed29916aa44a47f53"} Oct 30 06:30:05.285751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.285729 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" event={"ID":"f6f91032-e116-4d28-bc15-a4b5144777da","Type":"ContainerStarted","Data":"2af94a9fa754dae891e6e1338ca049b80e03c10e37e5c45d7284b0c5a378653a"} Oct 30 06:30:05.285919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.285902 5001 scope.go:122] "RemoveContainer" containerID="2af94a9fa754dae891e6e1338ca049b80e03c10e37e5c45d7284b0c5a378653a" Oct 30 06:30:05.286140 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:05.286119 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889030-f8mcq_odh-ai-gateway-infra(f6f91032-e116-4d28-bc15-a4b5144777da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" Oct 30 06:30:05.288186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.288107 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" event={"ID":"3f5393b2-ac60-4342-8185-8f77a0212db0","Type":"ContainerStarted","Data":"80e7e67751fe9999523b55f848c8c4b165fed3b67199fb142ac2024d64985be0"} Oct 30 06:30:05.290911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.290166 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" event={"ID":"2a0a42e5-b9fc-4eff-b9da-4969663d6027","Type":"ContainerStarted","Data":"949e578e309dada2b983e7a34f50c5ef4b36d46dcf784289bc22a48512c41ba5"} Oct 30 06:30:05.357978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.357928 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" podStartSLOduration=5.357910931 podStartE2EDuration="5.357910931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:05.357730339 +0000 UTC m=+2219.353687353" watchObservedRunningTime="2026-10-30 06:30:05.357910931 +0000 UTC m=+2219.353867943" Oct 30 06:30:05.413067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.413013 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" podStartSLOduration=5.412992398 podStartE2EDuration="5.412992398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:05.405622702 +0000 UTC m=+2219.401579724" watchObservedRunningTime="2026-10-30 06:30:05.412992398 +0000 UTC m=+2219.408949409" Oct 30 06:30:05.543819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.543753 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" podStartSLOduration=5.543735505 podStartE2EDuration="5.543735505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:05.540970875 +0000 UTC m=+2219.536927886" watchObservedRunningTime="2026-10-30 06:30:05.543735505 +0000 UTC m=+2219.539692516" Oct 30 06:30:05.625819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:05.625566 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" podStartSLOduration=5.625548589 podStartE2EDuration="5.625548589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:05.623814013 +0000 UTC m=+2219.619771058" watchObservedRunningTime="2026-10-30 06:30:05.625548589 +0000 UTC m=+2219.621505601" Oct 30 06:30:06.141681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.141645 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe11027-c91f-42bf-91f9-35c15a23da51" path="/var/lib/kubelet/pods/ebe11027-c91f-42bf-91f9-35c15a23da51/volumes" Oct 30 06:30:06.303282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.303248 5001 generic.go:356] "Generic (PLEG): container finished" podID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerID="80e7e67751fe9999523b55f848c8c4b165fed3b67199fb142ac2024d64985be0" exitCode=60 Oct 30 06:30:06.303791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.303356 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" event={"ID":"3f5393b2-ac60-4342-8185-8f77a0212db0","Type":"ContainerDied","Data":"80e7e67751fe9999523b55f848c8c4b165fed3b67199fb142ac2024d64985be0"} Oct 30 06:30:06.303791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.303432 5001 scope.go:122] "RemoveContainer" containerID="a1de637d0f7f03e14b581003e46f43c718ba3eadeee90f60ffbe514bfaa016a2" Oct 30 06:30:06.303913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.303789 5001 scope.go:122] "RemoveContainer" containerID="80e7e67751fe9999523b55f848c8c4b165fed3b67199fb142ac2024d64985be0" Oct 30 06:30:06.304175 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:06.304153 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4_odh-ai-gateway-infra(3f5393b2-ac60-4342-8185-8f77a0212db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" Oct 30 06:30:06.306151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.306126 5001 generic.go:356] "Generic (PLEG): container finished" podID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerID="949e578e309dada2b983e7a34f50c5ef4b36d46dcf784289bc22a48512c41ba5" exitCode=60 Oct 30 06:30:06.306275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.306204 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" event={"ID":"2a0a42e5-b9fc-4eff-b9da-4969663d6027","Type":"ContainerDied","Data":"949e578e309dada2b983e7a34f50c5ef4b36d46dcf784289bc22a48512c41ba5"} Oct 30 06:30:06.306521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.306503 5001 scope.go:122] "RemoveContainer" containerID="949e578e309dada2b983e7a34f50c5ef4b36d46dcf784289bc22a48512c41ba5" Oct 30 06:30:06.306892 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:06.306868 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq_odh-ai-gateway-infra(2a0a42e5-b9fc-4eff-b9da-4969663d6027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" Oct 30 06:30:06.308411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.308373 5001 generic.go:356] "Generic (PLEG): container finished" podID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerID="b5cc5c401c5bfefa18ce44f74d3a32bde96f42c5172299b86559ee7427ce6dc6" exitCode=60 Oct 30 06:30:06.308511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.308449 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" event={"ID":"4ac79889-b8fc-40c1-a988-946e1525dd2a","Type":"ContainerDied","Data":"b5cc5c401c5bfefa18ce44f74d3a32bde96f42c5172299b86559ee7427ce6dc6"} Oct 30 06:30:06.308683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.308669 5001 scope.go:122] "RemoveContainer" containerID="b5cc5c401c5bfefa18ce44f74d3a32bde96f42c5172299b86559ee7427ce6dc6" Oct 30 06:30:06.309010 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:06.308979 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx_odh-ai-gateway-infra(4ac79889-b8fc-40c1-a988-946e1525dd2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" Oct 30 06:30:06.310395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.310374 5001 generic.go:356] "Generic (PLEG): container finished" podID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerID="10a94a1738b598a7f2add23a59a35871dff7b36b2c525e12427f1842c7461930" exitCode=60 Oct 30 06:30:06.310504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.310448 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" event={"ID":"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6","Type":"ContainerDied","Data":"10a94a1738b598a7f2add23a59a35871dff7b36b2c525e12427f1842c7461930"} Oct 30 06:30:06.310719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.310702 5001 scope.go:122] "RemoveContainer" containerID="10a94a1738b598a7f2add23a59a35871dff7b36b2c525e12427f1842c7461930" Oct 30 06:30:06.311048 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:06.311022 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr_odh-ai-gateway-infra(5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" Oct 30 06:30:06.312647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.312626 5001 generic.go:356] "Generic (PLEG): container finished" podID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerID="c1e6df6f71a347a9c96821cc3325d9098b00a2411662796b8b606a686b384bda" exitCode=60 Oct 30 06:30:06.312776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.312758 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" event={"ID":"1f0799a8-e504-4570-ba51-1b2aadcbfd1f","Type":"ContainerDied","Data":"c1e6df6f71a347a9c96821cc3325d9098b00a2411662796b8b606a686b384bda"} Oct 30 06:30:06.313014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.312999 5001 scope.go:122] "RemoveContainer" containerID="c1e6df6f71a347a9c96821cc3325d9098b00a2411662796b8b606a686b384bda" Oct 30 06:30:06.313294 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:06.313275 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6_odh-ai-gateway-infra(1f0799a8-e504-4570-ba51-1b2aadcbfd1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" Oct 30 06:30:06.314967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.314947 5001 generic.go:356] "Generic (PLEG): container finished" podID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerID="c30e6bc92024f442735467c8568c75dfcf2a4066e21b4ae8160c1ff197e8c9f9" exitCode=60 Oct 30 06:30:06.315062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.314997 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" event={"ID":"78c12b75-839e-427e-8cb2-9d3873ea8ee4","Type":"ContainerDied","Data":"c30e6bc92024f442735467c8568c75dfcf2a4066e21b4ae8160c1ff197e8c9f9"} Oct 30 06:30:06.315260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.315244 5001 scope.go:122] "RemoveContainer" containerID="c30e6bc92024f442735467c8568c75dfcf2a4066e21b4ae8160c1ff197e8c9f9" Oct 30 06:30:06.315576 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:06.315552 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq_odh-ai-gateway-infra(78c12b75-839e-427e-8cb2-9d3873ea8ee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" Oct 30 06:30:06.317224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.317206 5001 scope.go:122] "RemoveContainer" containerID="473c6f2b1016998acd2d3cc90be7eba6846f3e6e9f11fbede8717a4446f849ed" Oct 30 06:30:06.317524 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:06.317501 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj_odh-ai-gateway-infra(a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" Oct 30 06:30:06.318995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.318974 5001 generic.go:356] "Generic (PLEG): container finished" podID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerID="403c5f8865d3c446f48e3d3635aed23bd2df68c750c17c1ed29916aa44a47f53" exitCode=60 Oct 30 06:30:06.319096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.319049 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" event={"ID":"6a0ea84e-5daf-494f-b5e6-be8757eccb1b","Type":"ContainerDied","Data":"403c5f8865d3c446f48e3d3635aed23bd2df68c750c17c1ed29916aa44a47f53"} Oct 30 06:30:06.319341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.319283 5001 scope.go:122] "RemoveContainer" containerID="403c5f8865d3c446f48e3d3635aed23bd2df68c750c17c1ed29916aa44a47f53" Oct 30 06:30:06.319592 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:06.319569 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5_odh-ai-gateway-infra(6a0ea84e-5daf-494f-b5e6-be8757eccb1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" Oct 30 06:30:06.321426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.321404 5001 generic.go:356] "Generic (PLEG): container finished" podID="f6f91032-e116-4d28-bc15-a4b5144777da" containerID="2af94a9fa754dae891e6e1338ca049b80e03c10e37e5c45d7284b0c5a378653a" exitCode=60 Oct 30 06:30:06.321506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.321457 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" event={"ID":"f6f91032-e116-4d28-bc15-a4b5144777da","Type":"ContainerDied","Data":"2af94a9fa754dae891e6e1338ca049b80e03c10e37e5c45d7284b0c5a378653a"} Oct 30 06:30:06.321746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.321727 5001 scope.go:122] "RemoveContainer" containerID="2af94a9fa754dae891e6e1338ca049b80e03c10e37e5c45d7284b0c5a378653a" Oct 30 06:30:06.322024 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:06.322003 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889030-f8mcq_odh-ai-gateway-infra(f6f91032-e116-4d28-bc15-a4b5144777da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" Oct 30 06:30:06.322390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.322368 5001 scope.go:122] "RemoveContainer" containerID="fd41ddfb8625fa5aa6d857327049d930e3c586cdd1d96729bfce987134989b14" Oct 30 06:30:06.337506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.337485 5001 scope.go:122] "RemoveContainer" containerID="3910a769a769d8d928062f8ce2a699480c3c75324ca7d91cdfb1dc581958673c" Oct 30 06:30:06.353803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.353777 5001 scope.go:122] "RemoveContainer" containerID="febbf1684fa6988bb9fc9367649ce38f09c3ef2b939ad07d47e1d8bf33532d9f" Oct 30 06:30:06.372461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.372434 5001 scope.go:122] "RemoveContainer" containerID="5b518f8e4a7c0cc4f579c07810eba762fa72e757fe695364c126c26322709d67" Oct 30 06:30:06.388387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.388367 5001 scope.go:122] "RemoveContainer" containerID="91ff4a304ffa78284c60e0e84c5694db5b15b8a98314541cf44781b75c8a0f62" Oct 30 06:30:06.404253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.404228 5001 scope.go:122] "RemoveContainer" containerID="793e0ad298e6ddaa6f7def81ba16750427daee80b0e076a0b6f97aa11c65c01e" Oct 30 06:30:06.419346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:06.419319 5001 scope.go:122] "RemoveContainer" containerID="ec0ec08df473ecf884fd6fb4376cc37ce054f1ea6e06756a6c1da8b0d86927b6" Oct 30 06:30:07.332954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:07.332924 5001 scope.go:122] "RemoveContainer" containerID="2af94a9fa754dae891e6e1338ca049b80e03c10e37e5c45d7284b0c5a378653a" Oct 30 06:30:07.333506 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:07.333339 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889030-f8mcq_odh-ai-gateway-infra(f6f91032-e116-4d28-bc15-a4b5144777da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" Oct 30 06:30:07.334816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:07.334796 5001 scope.go:122] "RemoveContainer" containerID="80e7e67751fe9999523b55f848c8c4b165fed3b67199fb142ac2024d64985be0" Oct 30 06:30:07.335173 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:07.335151 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4_odh-ai-gateway-infra(3f5393b2-ac60-4342-8185-8f77a0212db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" Oct 30 06:30:07.336578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:07.336561 5001 scope.go:122] "RemoveContainer" containerID="949e578e309dada2b983e7a34f50c5ef4b36d46dcf784289bc22a48512c41ba5" Oct 30 06:30:07.336855 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:07.336831 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq_odh-ai-gateway-infra(2a0a42e5-b9fc-4eff-b9da-4969663d6027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" Oct 30 06:30:07.338082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:07.338065 5001 scope.go:122] "RemoveContainer" containerID="b5cc5c401c5bfefa18ce44f74d3a32bde96f42c5172299b86559ee7427ce6dc6" Oct 30 06:30:07.338343 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:07.338318 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx_odh-ai-gateway-infra(4ac79889-b8fc-40c1-a988-946e1525dd2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" Oct 30 06:30:07.339782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:07.339763 5001 scope.go:122] "RemoveContainer" containerID="10a94a1738b598a7f2add23a59a35871dff7b36b2c525e12427f1842c7461930" Oct 30 06:30:07.340030 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:07.340007 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr_odh-ai-gateway-infra(5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" Oct 30 06:30:07.341511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:07.341493 5001 scope.go:122] "RemoveContainer" containerID="c1e6df6f71a347a9c96821cc3325d9098b00a2411662796b8b606a686b384bda" Oct 30 06:30:07.341716 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:07.341697 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6_odh-ai-gateway-infra(1f0799a8-e504-4570-ba51-1b2aadcbfd1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" Oct 30 06:30:07.343100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:07.343083 5001 scope.go:122] "RemoveContainer" containerID="c30e6bc92024f442735467c8568c75dfcf2a4066e21b4ae8160c1ff197e8c9f9" Oct 30 06:30:07.343299 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:07.343279 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq_odh-ai-gateway-infra(78c12b75-839e-427e-8cb2-9d3873ea8ee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" Oct 30 06:30:07.344954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:07.344938 5001 scope.go:122] "RemoveContainer" containerID="403c5f8865d3c446f48e3d3635aed23bd2df68c750c17c1ed29916aa44a47f53" Oct 30 06:30:07.345160 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:07.345139 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5_odh-ai-gateway-infra(6a0ea84e-5daf-494f-b5e6-be8757eccb1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" Oct 30 06:30:08.271178 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:08.271138 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:30:09.281893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:09.281859 5001 scope.go:122] "RemoveContainer" containerID="899a50b95f0e71d909e1f8cf463c3045883f4231f2a562a2e893b2dfde745c91" Oct 30 06:30:09.296317 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:09.296282 5001 scope.go:122] "RemoveContainer" containerID="1b72ddff188f7147eada9bc42eb41183f3ebba0b0c8de935243a32820dd9d089" Oct 30 06:30:09.307325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:09.307278 5001 scope.go:122] "RemoveContainer" containerID="3f7af04cc1abf351ab09a2162d0353e41ce3e2e5200dc8384ab7f70caa8cd3e5" Oct 30 06:30:09.317822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:09.317800 5001 scope.go:122] "RemoveContainer" containerID="2846c490b1bacd7d33b4efb866cdfc0674e2fb44477773e407fb8b77df6e1541" Oct 30 06:30:09.328359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:09.328341 5001 scope.go:122] "RemoveContainer" containerID="f8a8dc257d324a47bb2209c41250ffc8fdc0fe9189b0923eb654ac39003ea39b" Oct 30 06:30:11.572323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:11.568638 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9"] Oct 30 06:30:11.572323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:11.568883 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" containerID="cri-o://686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927" gracePeriod=30 Oct 30 06:30:12.659663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.659628 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b"] Oct 30 06:30:12.670020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.669993 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.705582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.705546 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b"] Oct 30 06:30:12.709384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.709344 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-b8b590-openshift-default-dockercfg-6ssjs\"" Oct 30 06:30:12.771249 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.771216 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmkgp\" (UniqueName: \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-kube-api-access-dmkgp\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.771496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.771284 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-podinfo\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.771496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.771469 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-credential-socket\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.771610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.771519 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69144896-68b9-4b57-9e4c-8bca3f96c640-istiod-ca-cert\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.771653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.771614 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-token\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.771697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.771665 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-socket\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.771777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.771764 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-envoy\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.771814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.771788 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-certs\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.771864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.771832 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-data\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.878343 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-data\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.878515 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmkgp\" (UniqueName: \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-kube-api-access-dmkgp\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.878583 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-podinfo\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.878664 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-credential-socket\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.878705 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69144896-68b9-4b57-9e4c-8bca3f96c640-istiod-ca-cert\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.878733 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-token\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.878786 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-socket\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.878857 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-envoy\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.878887 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-certs\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.879508 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-certs\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.879770 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-data\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.880962 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-credential-socket\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.881592 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69144896-68b9-4b57-9e4c-8bca3f96c640-istiod-ca-cert\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.882334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.881848 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-socket\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.939383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.925913 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmkgp\" (UniqueName: \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-kube-api-access-dmkgp\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.939383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.937247 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-envoy\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.957328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.951967 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-podinfo\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:12.981332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:12.976912 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-token\") pod \"e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:13.016330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:13.015615 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:13.453078 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:13.453041 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b"] Oct 30 06:30:13.455765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:13.455728 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:30:13.455871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:13.455796 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:30:13.455871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:13.455824 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:30:14.033484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:14.033451 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns"] Oct 30 06:30:14.045357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:14.045326 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75713d4c-x5fns"] Oct 30 06:30:14.119611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:14.119577 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="843533bb-ca51-450c-b95d-557df4ee6423" path="/var/lib/kubelet/pods/843533bb-ca51-450c-b95d-557df4ee6423/volumes" Oct 30 06:30:14.438528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:14.438490 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" event={"ID":"69144896-68b9-4b57-9e4c-8bca3f96c640","Type":"ContainerStarted","Data":"1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1"} Oct 30 06:30:14.438720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:14.438548 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" event={"ID":"69144896-68b9-4b57-9e4c-8bca3f96c640","Type":"ContainerStarted","Data":"9632b9ec20c7e28ad7bb3c17a6b571a6a76b7614b5a24e92a36241bccf2c6b95"} Oct 30 06:30:14.490167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:14.490106 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" podStartSLOduration=2.490085112 podStartE2EDuration="2.490085112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:14.482153169 +0000 UTC m=+2228.478110180" watchObservedRunningTime="2026-10-30 06:30:14.490085112 +0000 UTC m=+2228.486042123" Oct 30 06:30:15.017051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:15.017011 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:15.023955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:15.023927 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:15.037574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:15.037541 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs"] Oct 30 06:30:15.049187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:15.049153 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784206f4-hqmhs"] Oct 30 06:30:15.448191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:15.448159 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:15.449169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:15.449142 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:16.115166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:16.115137 5001 scope.go:122] "RemoveContainer" containerID="80e7e67751fe9999523b55f848c8c4b165fed3b67199fb142ac2024d64985be0" Oct 30 06:30:16.115671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:16.115231 5001 scope.go:122] "RemoveContainer" containerID="10a94a1738b598a7f2add23a59a35871dff7b36b2c525e12427f1842c7461930" Oct 30 06:30:16.115671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:16.115384 5001 scope.go:122] "RemoveContainer" containerID="473c6f2b1016998acd2d3cc90be7eba6846f3e6e9f11fbede8717a4446f849ed" Oct 30 06:30:16.121513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:16.121489 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5624a5-bb95-4c59-b1d4-0cf4d60443eb" path="/var/lib/kubelet/pods/ac5624a5-bb95-4c59-b1d4-0cf4d60443eb/volumes" Oct 30 06:30:16.376849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:16.376765 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:30:16.912433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:16.912334 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:30:16.996333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:16.990601 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-credential-socket\") pod \"d18bb796-a33f-41bc-93f2-302c700fd101\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " Oct 30 06:30:16.996333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:16.990650 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-certs\") pod \"d18bb796-a33f-41bc-93f2-302c700fd101\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " Oct 30 06:30:16.996333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:16.990713 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-kube-api-access-nd7vk\" (UniqueName: \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-kube-api-access-nd7vk\") pod \"d18bb796-a33f-41bc-93f2-302c700fd101\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " Oct 30 06:30:16.996333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:16.990771 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-socket\") pod \"d18bb796-a33f-41bc-93f2-302c700fd101\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " Oct 30 06:30:17.037859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.034270 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-socket" pod "d18bb796-a33f-41bc-93f2-302c700fd101" (UID: "d18bb796-a33f-41bc-93f2-302c700fd101"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:17.037859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.034688 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-certs" pod "d18bb796-a33f-41bc-93f2-302c700fd101" (UID: "d18bb796-a33f-41bc-93f2-302c700fd101"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:17.051751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.049531 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-credential-socket" pod "d18bb796-a33f-41bc-93f2-302c700fd101" (UID: "d18bb796-a33f-41bc-93f2-302c700fd101"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:17.051751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.049643 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d18bb796-a33f-41bc-93f2-302c700fd101-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18bb796-a33f-41bc-93f2-302c700fd101-istio-podinfo\") pod \"d18bb796-a33f-41bc-93f2-302c700fd101\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " Oct 30 06:30:17.051751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.049703 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-envoy\") pod \"d18bb796-a33f-41bc-93f2-302c700fd101\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " Oct 30 06:30:17.051751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.049803 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-data\") pod \"d18bb796-a33f-41bc-93f2-302c700fd101\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " Oct 30 06:30:17.051751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.049915 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-istio-token\" (UniqueName: \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-istio-token\") pod \"d18bb796-a33f-41bc-93f2-302c700fd101\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " Oct 30 06:30:17.051751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.050011 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d18bb796-a33f-41bc-93f2-302c700fd101-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18bb796-a33f-41bc-93f2-302c700fd101-istiod-ca-cert\") pod \"d18bb796-a33f-41bc-93f2-302c700fd101\" (UID: \"d18bb796-a33f-41bc-93f2-302c700fd101\") " Oct 30 06:30:17.051751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.050879 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:17.051751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.050906 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:17.051751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.050918 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:17.088633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.088274 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-data" pod "d18bb796-a33f-41bc-93f2-302c700fd101" (UID: "d18bb796-a33f-41bc-93f2-302c700fd101"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:17.105379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.103609 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18bb796-a33f-41bc-93f2-302c700fd101-istiod-ca-cert" pod "d18bb796-a33f-41bc-93f2-302c700fd101" (UID: "d18bb796-a33f-41bc-93f2-302c700fd101"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:17.112195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.111586 5001 scope.go:122] "RemoveContainer" containerID="c1e6df6f71a347a9c96821cc3325d9098b00a2411662796b8b606a686b384bda" Oct 30 06:30:17.121589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.121393 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-envoy" pod "d18bb796-a33f-41bc-93f2-302c700fd101" (UID: "d18bb796-a33f-41bc-93f2-302c700fd101"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:17.121589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.121552 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-kube-api-access-nd7vk" pod "d18bb796-a33f-41bc-93f2-302c700fd101" (UID: "d18bb796-a33f-41bc-93f2-302c700fd101"). InnerVolumeSpecName "kube-api-access-nd7vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:17.122021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.121671 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-istio-token" pod "d18bb796-a33f-41bc-93f2-302c700fd101" (UID: "d18bb796-a33f-41bc-93f2-302c700fd101"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:17.122376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.122288 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d18bb796-a33f-41bc-93f2-302c700fd101-istio-podinfo" pod "d18bb796-a33f-41bc-93f2-302c700fd101" (UID: "d18bb796-a33f-41bc-93f2-302c700fd101"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:30:17.159138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.159102 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd7vk\" (UniqueName: \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-kube-api-access-nd7vk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:17.159359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.159345 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18bb796-a33f-41bc-93f2-302c700fd101-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:17.159474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.159464 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:17.159561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.159550 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18bb796-a33f-41bc-93f2-302c700fd101-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:17.159653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.159643 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18bb796-a33f-41bc-93f2-302c700fd101-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:17.159770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.159759 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18bb796-a33f-41bc-93f2-302c700fd101-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:17.594618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.594580 5001 generic.go:356] "Generic (PLEG): container finished" podID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerID="6afa466f3c7af028d07525e2a8b2dff928d7759e653f5e8cf2e29753c1a778cc" exitCode=60 Oct 30 06:30:17.594960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.594931 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" event={"ID":"3f5393b2-ac60-4342-8185-8f77a0212db0","Type":"ContainerDied","Data":"6afa466f3c7af028d07525e2a8b2dff928d7759e653f5e8cf2e29753c1a778cc"} Oct 30 06:30:17.595684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.595660 5001 scope.go:122] "RemoveContainer" containerID="6afa466f3c7af028d07525e2a8b2dff928d7759e653f5e8cf2e29753c1a778cc" Oct 30 06:30:17.601367 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:17.601336 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4_odh-ai-gateway-infra(3f5393b2-ac60-4342-8185-8f77a0212db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" Oct 30 06:30:17.601876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.601859 5001 scope.go:122] "RemoveContainer" containerID="80e7e67751fe9999523b55f848c8c4b165fed3b67199fb142ac2024d64985be0" Oct 30 06:30:17.631234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.629553 5001 generic.go:356] "Generic (PLEG): container finished" podID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerID="5f14813ad037ed3a3879cd74a7aef53d85c60f9025ac7defc047ea5720eb3d89" exitCode=60 Oct 30 06:30:17.631234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.629831 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" event={"ID":"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6","Type":"ContainerDied","Data":"5f14813ad037ed3a3879cd74a7aef53d85c60f9025ac7defc047ea5720eb3d89"} Oct 30 06:30:17.631234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.630354 5001 scope.go:122] "RemoveContainer" containerID="5f14813ad037ed3a3879cd74a7aef53d85c60f9025ac7defc047ea5720eb3d89" Oct 30 06:30:17.631234 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:17.630855 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr_odh-ai-gateway-infra(5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" Oct 30 06:30:17.738045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.737543 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" event={"ID":"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb","Type":"ContainerStarted","Data":"1b9355556df097aabd6c91e3c49730bf188b67f358ea25e3457a4066dfd12ccc"} Oct 30 06:30:17.738753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.738730 5001 scope.go:122] "RemoveContainer" containerID="1b9355556df097aabd6c91e3c49730bf188b67f358ea25e3457a4066dfd12ccc" Oct 30 06:30:17.739279 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:17.739235 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj_odh-ai-gateway-infra(a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" Oct 30 06:30:17.741159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.741079 5001 generic.go:356] "Generic (PLEG): container finished" podID="d18bb796-a33f-41bc-93f2-302c700fd101" containerID="686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927" exitCode=0 Oct 30 06:30:17.742059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.742041 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" Oct 30 06:30:17.742440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.742416 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" event={"ID":"d18bb796-a33f-41bc-93f2-302c700fd101","Type":"ContainerDied","Data":"686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927"} Oct 30 06:30:17.742493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.742473 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9" event={"ID":"d18bb796-a33f-41bc-93f2-302c700fd101","Type":"ContainerDied","Data":"dba2d59d3b397cf2e52df92f5009258b8bf8444e61f4ae7641de9109568d7380"} Oct 30 06:30:17.745128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.745106 5001 scope.go:122] "RemoveContainer" containerID="10a94a1738b598a7f2add23a59a35871dff7b36b2c525e12427f1842c7461930" Oct 30 06:30:17.768959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.768935 5001 scope.go:122] "RemoveContainer" containerID="686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927" Oct 30 06:30:17.783296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.783270 5001 scope.go:122] "RemoveContainer" containerID="2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b" Oct 30 06:30:17.796501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.796478 5001 scope.go:122] "RemoveContainer" containerID="686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927" Oct 30 06:30:17.796843 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:17.796821 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927\": container with ID starting with 686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927 not found: ID does not exist" containerID="686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927" Oct 30 06:30:17.796936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.796856 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927"} err="failed to get container status \"686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927\": rpc error: code = NotFound desc = could not find container \"686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927\": container with ID starting with 686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927 not found: ID does not exist" Oct 30 06:30:17.796936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.796883 5001 scope.go:122] "RemoveContainer" containerID="2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b" Oct 30 06:30:17.797235 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:17.797218 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b\": container with ID starting with 2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b not found: ID does not exist" containerID="2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b" Oct 30 06:30:17.797274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.797243 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b"} err="failed to get container status \"2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b\": rpc error: code = NotFound desc = could not find container \"2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b\": container with ID starting with 2fb9a3bffb8f6dd541df3c2477648b867c3d5f02e1e3bf8524e144cb7d24d84b not found: ID does not exist" Oct 30 06:30:17.825545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.825515 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9"] Oct 30 06:30:17.835240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:17.835212 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-3bbfc8-openshift-default-6cfd58d8d5-752v9"] Oct 30 06:30:18.110574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.110544 5001 scope.go:122] "RemoveContainer" containerID="b5cc5c401c5bfefa18ce44f74d3a32bde96f42c5172299b86559ee7427ce6dc6" Oct 30 06:30:18.110744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.110658 5001 scope.go:122] "RemoveContainer" containerID="2af94a9fa754dae891e6e1338ca049b80e03c10e37e5c45d7284b0c5a378653a" Oct 30 06:30:18.121645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.121617 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" path="/var/lib/kubelet/pods/d18bb796-a33f-41bc-93f2-302c700fd101/volumes" Oct 30 06:30:18.659179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.659149 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4"] Oct 30 06:30:18.711516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.711486 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr"] Oct 30 06:30:18.769330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.756225 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" event={"ID":"f6f91032-e116-4d28-bc15-a4b5144777da","Type":"ContainerStarted","Data":"f39727d4a98ae5c1dd2be1f46064a9e0f8b7c3395783ca9c410b1888dee73c5b"} Oct 30 06:30:18.769330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.762206 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" event={"ID":"4ac79889-b8fc-40c1-a988-946e1525dd2a","Type":"ContainerStarted","Data":"aceef4f1f58d2c550571185fb1a6c6cb1ffad19707fee2a13860d385af0a572a"} Oct 30 06:30:18.769330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.766717 5001 generic.go:356] "Generic (PLEG): container finished" podID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerID="502be3677507fcb010c8da379383556f8bf22a792465ba96afe7be9ca3833845" exitCode=60 Oct 30 06:30:18.769330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.766789 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" event={"ID":"1f0799a8-e504-4570-ba51-1b2aadcbfd1f","Type":"ContainerDied","Data":"502be3677507fcb010c8da379383556f8bf22a792465ba96afe7be9ca3833845"} Oct 30 06:30:18.769330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.766855 5001 scope.go:122] "RemoveContainer" containerID="c1e6df6f71a347a9c96821cc3325d9098b00a2411662796b8b606a686b384bda" Oct 30 06:30:18.769330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.767292 5001 scope.go:122] "RemoveContainer" containerID="502be3677507fcb010c8da379383556f8bf22a792465ba96afe7be9ca3833845" Oct 30 06:30:18.769330 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:18.767708 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6_odh-ai-gateway-infra(1f0799a8-e504-4570-ba51-1b2aadcbfd1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" Oct 30 06:30:18.771022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.770281 5001 generic.go:356] "Generic (PLEG): container finished" podID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerID="1b9355556df097aabd6c91e3c49730bf188b67f358ea25e3457a4066dfd12ccc" exitCode=60 Oct 30 06:30:18.771022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.770334 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" event={"ID":"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb","Type":"ContainerDied","Data":"1b9355556df097aabd6c91e3c49730bf188b67f358ea25e3457a4066dfd12ccc"} Oct 30 06:30:18.771022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.770684 5001 scope.go:122] "RemoveContainer" containerID="1b9355556df097aabd6c91e3c49730bf188b67f358ea25e3457a4066dfd12ccc" Oct 30 06:30:18.771022 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:18.770984 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj_odh-ai-gateway-infra(a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" Oct 30 06:30:18.786760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.786733 5001 scope.go:122] "RemoveContainer" containerID="473c6f2b1016998acd2d3cc90be7eba6846f3e6e9f11fbede8717a4446f849ed" Oct 30 06:30:18.807351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.806041 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj"] Oct 30 06:30:18.807351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.806469 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" podStartSLOduration=18.806455683 podStartE2EDuration="18.806455683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:18.797679532 +0000 UTC m=+2232.793636542" watchObservedRunningTime="2026-10-30 06:30:18.806455683 +0000 UTC m=+2232.802412692" Oct 30 06:30:18.980617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:18.980591 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" Oct 30 06:30:19.015106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.015081 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5393b2-ac60-4342-8185-8f77a0212db0-kube-api-access-9jhzr\" (UniqueName: \"kubernetes.io/projected/3f5393b2-ac60-4342-8185-8f77a0212db0-kube-api-access-9jhzr\") pod \"3f5393b2-ac60-4342-8185-8f77a0212db0\" (UID: \"3f5393b2-ac60-4342-8185-8f77a0212db0\") " Oct 30 06:30:19.015253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.015237 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5393b2-ac60-4342-8185-8f77a0212db0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5393b2-ac60-4342-8185-8f77a0212db0-maas-api-service-ca\") pod \"3f5393b2-ac60-4342-8185-8f77a0212db0\" (UID: \"3f5393b2-ac60-4342-8185-8f77a0212db0\") " Oct 30 06:30:19.015974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.015820 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5393b2-ac60-4342-8185-8f77a0212db0-maas-api-service-ca" pod "3f5393b2-ac60-4342-8185-8f77a0212db0" (UID: "3f5393b2-ac60-4342-8185-8f77a0212db0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:19.016106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.015998 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5393b2-ac60-4342-8185-8f77a0212db0-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:19.017604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.017577 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5393b2-ac60-4342-8185-8f77a0212db0-kube-api-access-9jhzr" pod "3f5393b2-ac60-4342-8185-8f77a0212db0" (UID: "3f5393b2-ac60-4342-8185-8f77a0212db0"). InnerVolumeSpecName "kube-api-access-9jhzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:19.027516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.027494 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" Oct 30 06:30:19.037557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.037525 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj"] Oct 30 06:30:19.042191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.042169 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f3c7a3-zdrzj"] Oct 30 06:30:19.125658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.125622 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jhzr\" (UniqueName: \"kubernetes.io/projected/3f5393b2-ac60-4342-8185-8f77a0212db0-kube-api-access-9jhzr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:19.168071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.166211 5001 scope.go:122] "RemoveContainer" containerID="403c5f8865d3c446f48e3d3635aed23bd2df68c750c17c1ed29916aa44a47f53" Oct 30 06:30:19.168071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.167201 5001 scope.go:122] "RemoveContainer" containerID="c30e6bc92024f442735467c8568c75dfcf2a4066e21b4ae8160c1ff197e8c9f9" Oct 30 06:30:19.227070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.227042 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-maas-api-service-ca\") pod \"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6\" (UID: \"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6\") " Oct 30 06:30:19.227201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.227100 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-kube-api-access-tk2tz\" (UniqueName: \"kubernetes.io/projected/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-kube-api-access-tk2tz\") pod \"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6\" (UID: \"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6\") " Oct 30 06:30:19.227474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.227448 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-maas-api-service-ca" pod "5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" (UID: "5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:19.228769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.228743 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:19.229843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.229818 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-kube-api-access-tk2tz" pod "5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" (UID: "5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6"). InnerVolumeSpecName "kube-api-access-tk2tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:19.329784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.329754 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk2tz\" (UniqueName: \"kubernetes.io/projected/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6-kube-api-access-tk2tz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:19.793658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.793618 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" event={"ID":"78c12b75-839e-427e-8cb2-9d3873ea8ee4","Type":"ContainerStarted","Data":"f38d6761890f87b039f987b3b821377a79ca410e876327d7e2248f47724a424b"} Oct 30 06:30:19.797728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.797702 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" event={"ID":"6a0ea84e-5daf-494f-b5e6-be8757eccb1b","Type":"ContainerStarted","Data":"9a8d0b188ace6f285c902117946eb7b59311dbaa097843a9fc0765e619ac3787"} Oct 30 06:30:19.799947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.799924 5001 generic.go:356] "Generic (PLEG): container finished" podID="f6f91032-e116-4d28-bc15-a4b5144777da" containerID="f39727d4a98ae5c1dd2be1f46064a9e0f8b7c3395783ca9c410b1888dee73c5b" exitCode=60 Oct 30 06:30:19.800054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.799994 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" event={"ID":"f6f91032-e116-4d28-bc15-a4b5144777da","Type":"ContainerDied","Data":"f39727d4a98ae5c1dd2be1f46064a9e0f8b7c3395783ca9c410b1888dee73c5b"} Oct 30 06:30:19.800054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.800042 5001 scope.go:122] "RemoveContainer" containerID="2af94a9fa754dae891e6e1338ca049b80e03c10e37e5c45d7284b0c5a378653a" Oct 30 06:30:19.800430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.800415 5001 scope.go:122] "RemoveContainer" containerID="f39727d4a98ae5c1dd2be1f46064a9e0f8b7c3395783ca9c410b1888dee73c5b" Oct 30 06:30:19.800774 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:19.800740 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889030-f8mcq_odh-ai-gateway-infra(f6f91032-e116-4d28-bc15-a4b5144777da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" Oct 30 06:30:19.802101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.802078 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" event={"ID":"3f5393b2-ac60-4342-8185-8f77a0212db0","Type":"ContainerDied","Data":"7963982d9a32b2c62f1198e076a401cc6cdbd50ddc859c0cb2e9a623a8864f3c"} Oct 30 06:30:19.802371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.802295 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4" Oct 30 06:30:19.814757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.814732 5001 generic.go:356] "Generic (PLEG): container finished" podID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerID="aceef4f1f58d2c550571185fb1a6c6cb1ffad19707fee2a13860d385af0a572a" exitCode=60 Oct 30 06:30:19.815082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.815059 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" event={"ID":"4ac79889-b8fc-40c1-a988-946e1525dd2a","Type":"ContainerDied","Data":"aceef4f1f58d2c550571185fb1a6c6cb1ffad19707fee2a13860d385af0a572a"} Oct 30 06:30:19.815727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.815696 5001 scope.go:122] "RemoveContainer" containerID="aceef4f1f58d2c550571185fb1a6c6cb1ffad19707fee2a13860d385af0a572a" Oct 30 06:30:19.816200 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:19.816178 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx_odh-ai-gateway-infra(4ac79889-b8fc-40c1-a988-946e1525dd2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" Oct 30 06:30:19.817976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.817952 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq"] Oct 30 06:30:19.818454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.818436 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" Oct 30 06:30:19.818543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.818440 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr" event={"ID":"5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6","Type":"ContainerDied","Data":"ecf8a6c5a7aef9da99d44a218d23353eb9480c22ae65c61ee7a2a50eef081b2f"} Oct 30 06:30:19.830489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.830465 5001 scope.go:122] "RemoveContainer" containerID="6afa466f3c7af028d07525e2a8b2dff928d7759e653f5e8cf2e29753c1a778cc" Oct 30 06:30:19.854041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.854010 5001 scope.go:122] "RemoveContainer" containerID="b5cc5c401c5bfefa18ce44f74d3a32bde96f42c5172299b86559ee7427ce6dc6" Oct 30 06:30:19.871715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.871691 5001 scope.go:122] "RemoveContainer" containerID="5f14813ad037ed3a3879cd74a7aef53d85c60f9025ac7defc047ea5720eb3d89" Oct 30 06:30:19.887641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.887596 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4"] Oct 30 06:30:19.903904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.900385 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889030-cvrl4"] Oct 30 06:30:19.904025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.903929 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6"] Oct 30 06:30:19.999448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:19.998424 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" Oct 30 06:30:20.024397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.022948 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx"] Oct 30 06:30:20.038319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.038259 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-kube-api-access-jq899\" (UniqueName: \"kubernetes.io/projected/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-kube-api-access-jq899\") pod \"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb\" (UID: \"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb\") " Oct 30 06:30:20.038561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.038538 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-maas-api-service-ca\") pod \"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb\" (UID: \"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb\") " Oct 30 06:30:20.039673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.039647 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-maas-api-service-ca" pod "a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" (UID: "a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:20.041877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.041849 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-kube-api-access-jq899" pod "a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" (UID: "a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb"). InnerVolumeSpecName "kube-api-access-jq899". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:20.070261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.070220 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr"] Oct 30 06:30:20.077945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.077916 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889030-r74nr"] Oct 30 06:30:20.100368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.100339 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" Oct 30 06:30:20.110505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.110482 5001 scope.go:122] "RemoveContainer" containerID="949e578e309dada2b983e7a34f50c5ef4b36d46dcf784289bc22a48512c41ba5" Oct 30 06:30:20.121599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.121573 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" path="/var/lib/kubelet/pods/3f5393b2-ac60-4342-8185-8f77a0212db0/volumes" Oct 30 06:30:20.121995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.121979 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" path="/var/lib/kubelet/pods/5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6/volumes" Oct 30 06:30:20.122448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.122433 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e5df40-aa16-4408-820e-11bcafd55ec0" path="/var/lib/kubelet/pods/f5e5df40-aa16-4408-820e-11bcafd55ec0/volumes" Oct 30 06:30:20.140156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.140125 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-kube-api-access-fblqk\" (UniqueName: \"kubernetes.io/projected/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-kube-api-access-fblqk\") pod \"1f0799a8-e504-4570-ba51-1b2aadcbfd1f\" (UID: \"1f0799a8-e504-4570-ba51-1b2aadcbfd1f\") " Oct 30 06:30:20.140600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.140296 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-maas-api-service-ca\") pod \"1f0799a8-e504-4570-ba51-1b2aadcbfd1f\" (UID: \"1f0799a8-e504-4570-ba51-1b2aadcbfd1f\") " Oct 30 06:30:20.140663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.140646 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-maas-api-service-ca" pod "1f0799a8-e504-4570-ba51-1b2aadcbfd1f" (UID: "1f0799a8-e504-4570-ba51-1b2aadcbfd1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:20.141483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.141420 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq899\" (UniqueName: \"kubernetes.io/projected/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-kube-api-access-jq899\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:20.141483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.141447 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:20.141483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.141461 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:20.143152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.143127 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-kube-api-access-fblqk" pod "1f0799a8-e504-4570-ba51-1b2aadcbfd1f" (UID: "1f0799a8-e504-4570-ba51-1b2aadcbfd1f"). InnerVolumeSpecName "kube-api-access-fblqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:20.242397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.242363 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fblqk\" (UniqueName: \"kubernetes.io/projected/1f0799a8-e504-4570-ba51-1b2aadcbfd1f-kube-api-access-fblqk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:20.836629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.836597 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" event={"ID":"2a0a42e5-b9fc-4eff-b9da-4969663d6027","Type":"ContainerStarted","Data":"1725b075ee92a77b184a8d7c0593a5f5754b1b307f42d086f0bf993aa463727b"} Oct 30 06:30:20.841039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.841011 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" Oct 30 06:30:20.841162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.841034 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6" event={"ID":"1f0799a8-e504-4570-ba51-1b2aadcbfd1f","Type":"ContainerDied","Data":"4c95af7c4c12621a6bc61e326bc2fdaf0bc0292cb5a55e0f584a59fe9163024f"} Oct 30 06:30:20.841162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.841101 5001 scope.go:122] "RemoveContainer" containerID="502be3677507fcb010c8da379383556f8bf22a792465ba96afe7be9ca3833845" Oct 30 06:30:20.844026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.844000 5001 generic.go:356] "Generic (PLEG): container finished" podID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerID="f38d6761890f87b039f987b3b821377a79ca410e876327d7e2248f47724a424b" exitCode=60 Oct 30 06:30:20.844140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.844109 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" event={"ID":"78c12b75-839e-427e-8cb2-9d3873ea8ee4","Type":"ContainerDied","Data":"f38d6761890f87b039f987b3b821377a79ca410e876327d7e2248f47724a424b"} Oct 30 06:30:20.844536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.844504 5001 scope.go:122] "RemoveContainer" containerID="f38d6761890f87b039f987b3b821377a79ca410e876327d7e2248f47724a424b" Oct 30 06:30:20.844813 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:20.844791 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq_odh-ai-gateway-infra(78c12b75-839e-427e-8cb2-9d3873ea8ee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" Oct 30 06:30:20.847574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.847549 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" Oct 30 06:30:20.847574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.847561 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj" event={"ID":"a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb","Type":"ContainerDied","Data":"9aa61823ec349a71f7b5222b4723f81ae099e155de0410266565320a8502f448"} Oct 30 06:30:20.854637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.850066 5001 generic.go:356] "Generic (PLEG): container finished" podID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerID="9a8d0b188ace6f285c902117946eb7b59311dbaa097843a9fc0765e619ac3787" exitCode=60 Oct 30 06:30:20.854637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.850132 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" event={"ID":"6a0ea84e-5daf-494f-b5e6-be8757eccb1b","Type":"ContainerDied","Data":"9a8d0b188ace6f285c902117946eb7b59311dbaa097843a9fc0765e619ac3787"} Oct 30 06:30:20.854637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.850482 5001 scope.go:122] "RemoveContainer" containerID="9a8d0b188ace6f285c902117946eb7b59311dbaa097843a9fc0765e619ac3787" Oct 30 06:30:20.854637 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:20.850805 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5_odh-ai-gateway-infra(6a0ea84e-5daf-494f-b5e6-be8757eccb1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" Oct 30 06:30:20.863532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.860394 5001 scope.go:122] "RemoveContainer" containerID="c30e6bc92024f442735467c8568c75dfcf2a4066e21b4ae8160c1ff197e8c9f9" Oct 30 06:30:20.863777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.863756 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq"] Oct 30 06:30:20.887157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.886465 5001 scope.go:122] "RemoveContainer" containerID="1b9355556df097aabd6c91e3c49730bf188b67f358ea25e3457a4066dfd12ccc" Oct 30 06:30:20.910987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.910954 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6"] Oct 30 06:30:20.912460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.912120 5001 scope.go:122] "RemoveContainer" containerID="403c5f8865d3c446f48e3d3635aed23bd2df68c750c17c1ed29916aa44a47f53" Oct 30 06:30:20.917462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.917442 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889030-w9vl6"] Oct 30 06:30:20.985290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:20.985244 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5"] Oct 30 06:30:21.157577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.157545 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj"] Oct 30 06:30:21.169970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.169853 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" Oct 30 06:30:21.170195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.170174 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889030-pzpdj"] Oct 30 06:30:21.176406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.176383 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" Oct 30 06:30:21.230697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.230662 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac79889-b8fc-40c1-a988-946e1525dd2a-kube-api-access-zc67p\" (UniqueName: \"kubernetes.io/projected/4ac79889-b8fc-40c1-a988-946e1525dd2a-kube-api-access-zc67p\") pod \"4ac79889-b8fc-40c1-a988-946e1525dd2a\" (UID: \"4ac79889-b8fc-40c1-a988-946e1525dd2a\") " Oct 30 06:30:21.230891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.230873 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f91032-e116-4d28-bc15-a4b5144777da-kube-api-access-krzgz\" (UniqueName: \"kubernetes.io/projected/f6f91032-e116-4d28-bc15-a4b5144777da-kube-api-access-krzgz\") pod \"f6f91032-e116-4d28-bc15-a4b5144777da\" (UID: \"f6f91032-e116-4d28-bc15-a4b5144777da\") " Oct 30 06:30:21.230947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.230928 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac79889-b8fc-40c1-a988-946e1525dd2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac79889-b8fc-40c1-a988-946e1525dd2a-maas-api-service-ca\") pod \"4ac79889-b8fc-40c1-a988-946e1525dd2a\" (UID: \"4ac79889-b8fc-40c1-a988-946e1525dd2a\") " Oct 30 06:30:21.231092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.231075 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f91032-e116-4d28-bc15-a4b5144777da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f91032-e116-4d28-bc15-a4b5144777da-maas-api-service-ca\") pod \"f6f91032-e116-4d28-bc15-a4b5144777da\" (UID: \"f6f91032-e116-4d28-bc15-a4b5144777da\") " Oct 30 06:30:21.231400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.231370 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac79889-b8fc-40c1-a988-946e1525dd2a-maas-api-service-ca" pod "4ac79889-b8fc-40c1-a988-946e1525dd2a" (UID: "4ac79889-b8fc-40c1-a988-946e1525dd2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:21.231805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.231742 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac79889-b8fc-40c1-a988-946e1525dd2a-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:21.232097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.232072 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f91032-e116-4d28-bc15-a4b5144777da-maas-api-service-ca" pod "f6f91032-e116-4d28-bc15-a4b5144777da" (UID: "f6f91032-e116-4d28-bc15-a4b5144777da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:21.233424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.233395 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f91032-e116-4d28-bc15-a4b5144777da-kube-api-access-krzgz" pod "f6f91032-e116-4d28-bc15-a4b5144777da" (UID: "f6f91032-e116-4d28-bc15-a4b5144777da"). InnerVolumeSpecName "kube-api-access-krzgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:21.235009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.234984 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac79889-b8fc-40c1-a988-946e1525dd2a-kube-api-access-zc67p" pod "4ac79889-b8fc-40c1-a988-946e1525dd2a" (UID: "4ac79889-b8fc-40c1-a988-946e1525dd2a"). InnerVolumeSpecName "kube-api-access-zc67p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:21.332800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.332769 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f91032-e116-4d28-bc15-a4b5144777da-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:21.332800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.332799 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc67p\" (UniqueName: \"kubernetes.io/projected/4ac79889-b8fc-40c1-a988-946e1525dd2a-kube-api-access-zc67p\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:21.333014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.332811 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krzgz\" (UniqueName: \"kubernetes.io/projected/f6f91032-e116-4d28-bc15-a4b5144777da-kube-api-access-krzgz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:21.867095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.867067 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" Oct 30 06:30:21.867255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.867062 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq" event={"ID":"f6f91032-e116-4d28-bc15-a4b5144777da","Type":"ContainerDied","Data":"cc423f27728a1d6ed2bc6ccd5a2b5e3dbeb9c2b04e807743184a71700692e19b"} Oct 30 06:30:21.867255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.867196 5001 scope.go:122] "RemoveContainer" containerID="f39727d4a98ae5c1dd2be1f46064a9e0f8b7c3395783ca9c410b1888dee73c5b" Oct 30 06:30:21.870208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.870182 5001 generic.go:356] "Generic (PLEG): container finished" podID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerID="1725b075ee92a77b184a8d7c0593a5f5754b1b307f42d086f0bf993aa463727b" exitCode=60 Oct 30 06:30:21.870716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.870267 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" event={"ID":"2a0a42e5-b9fc-4eff-b9da-4969663d6027","Type":"ContainerDied","Data":"1725b075ee92a77b184a8d7c0593a5f5754b1b307f42d086f0bf993aa463727b"} Oct 30 06:30:21.870864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.870846 5001 scope.go:122] "RemoveContainer" containerID="1725b075ee92a77b184a8d7c0593a5f5754b1b307f42d086f0bf993aa463727b" Oct 30 06:30:21.871231 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:21.871207 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq_odh-ai-gateway-infra(2a0a42e5-b9fc-4eff-b9da-4969663d6027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" Oct 30 06:30:21.872007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.871987 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" Oct 30 06:30:21.872110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.872002 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx" event={"ID":"4ac79889-b8fc-40c1-a988-946e1525dd2a","Type":"ContainerDied","Data":"b6bc6c1e3c06310171e3a01ac1ca39876b09cc1e974474eb43b5a217e1988996"} Oct 30 06:30:21.884650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.884625 5001 scope.go:122] "RemoveContainer" containerID="949e578e309dada2b983e7a34f50c5ef4b36d46dcf784289bc22a48512c41ba5" Oct 30 06:30:21.902491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.902464 5001 scope.go:122] "RemoveContainer" containerID="aceef4f1f58d2c550571185fb1a6c6cb1ffad19707fee2a13860d385af0a572a" Oct 30 06:30:21.962359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.960517 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx"] Oct 30 06:30:21.967924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.967837 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889030-cb7nx"] Oct 30 06:30:21.985492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.985454 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq"] Oct 30 06:30:21.997436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:21.991666 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889030-f8mcq"] Oct 30 06:30:22.008188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.008158 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq"] Oct 30 06:30:22.083642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.083615 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" Oct 30 06:30:22.100648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.100621 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" Oct 30 06:30:22.118654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.118597 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" path="/var/lib/kubelet/pods/1f0799a8-e504-4570-ba51-1b2aadcbfd1f/volumes" Oct 30 06:30:22.119074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.119061 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" path="/var/lib/kubelet/pods/4ac79889-b8fc-40c1-a988-946e1525dd2a/volumes" Oct 30 06:30:22.119509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.119495 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" path="/var/lib/kubelet/pods/a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb/volumes" Oct 30 06:30:22.120194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.120179 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" path="/var/lib/kubelet/pods/f6f91032-e116-4d28-bc15-a4b5144777da/volumes" Oct 30 06:30:22.146900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.146865 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-kube-api-access-qcbd5\" (UniqueName: \"kubernetes.io/projected/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-kube-api-access-qcbd5\") pod \"6a0ea84e-5daf-494f-b5e6-be8757eccb1b\" (UID: \"6a0ea84e-5daf-494f-b5e6-be8757eccb1b\") " Oct 30 06:30:22.147078 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.147018 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c12b75-839e-427e-8cb2-9d3873ea8ee4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c12b75-839e-427e-8cb2-9d3873ea8ee4-maas-api-service-ca\") pod \"78c12b75-839e-427e-8cb2-9d3873ea8ee4\" (UID: \"78c12b75-839e-427e-8cb2-9d3873ea8ee4\") " Oct 30 06:30:22.147150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.147086 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c12b75-839e-427e-8cb2-9d3873ea8ee4-kube-api-access-q5gq6\" (UniqueName: \"kubernetes.io/projected/78c12b75-839e-427e-8cb2-9d3873ea8ee4-kube-api-access-q5gq6\") pod \"78c12b75-839e-427e-8cb2-9d3873ea8ee4\" (UID: \"78c12b75-839e-427e-8cb2-9d3873ea8ee4\") " Oct 30 06:30:22.147242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.147218 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-maas-api-service-ca\") pod \"6a0ea84e-5daf-494f-b5e6-be8757eccb1b\" (UID: \"6a0ea84e-5daf-494f-b5e6-be8757eccb1b\") " Oct 30 06:30:22.147474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.147451 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c12b75-839e-427e-8cb2-9d3873ea8ee4-maas-api-service-ca" pod "78c12b75-839e-427e-8cb2-9d3873ea8ee4" (UID: "78c12b75-839e-427e-8cb2-9d3873ea8ee4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:22.147682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.147654 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-maas-api-service-ca" pod "6a0ea84e-5daf-494f-b5e6-be8757eccb1b" (UID: "6a0ea84e-5daf-494f-b5e6-be8757eccb1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:22.148250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.148226 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:22.148435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.148417 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c12b75-839e-427e-8cb2-9d3873ea8ee4-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:22.149684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.149663 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-kube-api-access-qcbd5" pod "6a0ea84e-5daf-494f-b5e6-be8757eccb1b" (UID: "6a0ea84e-5daf-494f-b5e6-be8757eccb1b"). InnerVolumeSpecName "kube-api-access-qcbd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:22.150267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.150247 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c12b75-839e-427e-8cb2-9d3873ea8ee4-kube-api-access-q5gq6" pod "78c12b75-839e-427e-8cb2-9d3873ea8ee4" (UID: "78c12b75-839e-427e-8cb2-9d3873ea8ee4"). InnerVolumeSpecName "kube-api-access-q5gq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:22.250506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.250465 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcbd5\" (UniqueName: \"kubernetes.io/projected/6a0ea84e-5daf-494f-b5e6-be8757eccb1b-kube-api-access-qcbd5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:22.250506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.250505 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5gq6\" (UniqueName: \"kubernetes.io/projected/78c12b75-839e-427e-8cb2-9d3873ea8ee4-kube-api-access-q5gq6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:22.935576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.935549 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" Oct 30 06:30:22.935576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.935557 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq" event={"ID":"78c12b75-839e-427e-8cb2-9d3873ea8ee4","Type":"ContainerDied","Data":"ab314e5ed407bd91d58882f479fdc643d61698b146bacc502f439e595e6a12fa"} Oct 30 06:30:22.935789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.935631 5001 scope.go:122] "RemoveContainer" containerID="f38d6761890f87b039f987b3b821377a79ca410e876327d7e2248f47724a424b" Oct 30 06:30:22.937594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.937563 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" event={"ID":"6a0ea84e-5daf-494f-b5e6-be8757eccb1b","Type":"ContainerDied","Data":"599c3717587b32299096e201ee530bed7e96dc856958cae2bd29f9afb90aedd2"} Oct 30 06:30:22.937786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.937650 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5" Oct 30 06:30:22.950535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:22.950516 5001 scope.go:122] "RemoveContainer" containerID="9a8d0b188ace6f285c902117946eb7b59311dbaa097843a9fc0765e619ac3787" Oct 30 06:30:23.014529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.014496 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5"] Oct 30 06:30:23.020292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.020266 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889030-vjsj5"] Oct 30 06:30:23.052568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.052224 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq"] Oct 30 06:30:23.060396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.059400 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889030-wx5sq"] Oct 30 06:30:23.119750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.119723 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" Oct 30 06:30:23.168789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.168756 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0a42e5-b9fc-4eff-b9da-4969663d6027-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0a42e5-b9fc-4eff-b9da-4969663d6027-maas-api-service-ca\") pod \"2a0a42e5-b9fc-4eff-b9da-4969663d6027\" (UID: \"2a0a42e5-b9fc-4eff-b9da-4969663d6027\") " Oct 30 06:30:23.169163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.169140 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0a42e5-b9fc-4eff-b9da-4969663d6027-kube-api-access-9pbkq\" (UniqueName: \"kubernetes.io/projected/2a0a42e5-b9fc-4eff-b9da-4969663d6027-kube-api-access-9pbkq\") pod \"2a0a42e5-b9fc-4eff-b9da-4969663d6027\" (UID: \"2a0a42e5-b9fc-4eff-b9da-4969663d6027\") " Oct 30 06:30:23.169279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.169182 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0a42e5-b9fc-4eff-b9da-4969663d6027-maas-api-service-ca" pod "2a0a42e5-b9fc-4eff-b9da-4969663d6027" (UID: "2a0a42e5-b9fc-4eff-b9da-4969663d6027"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:23.169835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.169810 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0a42e5-b9fc-4eff-b9da-4969663d6027-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:23.172131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.172096 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0a42e5-b9fc-4eff-b9da-4969663d6027-kube-api-access-9pbkq" pod "2a0a42e5-b9fc-4eff-b9da-4969663d6027" (UID: "2a0a42e5-b9fc-4eff-b9da-4969663d6027"). InnerVolumeSpecName "kube-api-access-9pbkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:23.271207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.271173 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pbkq\" (UniqueName: \"kubernetes.io/projected/2a0a42e5-b9fc-4eff-b9da-4969663d6027-kube-api-access-9pbkq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:23.950102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.950077 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" Oct 30 06:30:23.950263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.950073 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq" event={"ID":"2a0a42e5-b9fc-4eff-b9da-4969663d6027","Type":"ContainerDied","Data":"dd0771f76d511ee6079e45f22950dad76c1e161c4d8dba17007dd902e914372d"} Oct 30 06:30:23.950263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:23.950204 5001 scope.go:122] "RemoveContainer" containerID="1725b075ee92a77b184a8d7c0593a5f5754b1b307f42d086f0bf993aa463727b" Oct 30 06:30:24.004364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:24.004333 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq"] Oct 30 06:30:24.013187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:24.013160 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889030-qm4jq"] Oct 30 06:30:24.119036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:24.119002 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" path="/var/lib/kubelet/pods/2a0a42e5-b9fc-4eff-b9da-4969663d6027/volumes" Oct 30 06:30:24.119564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:24.119549 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" path="/var/lib/kubelet/pods/6a0ea84e-5daf-494f-b5e6-be8757eccb1b/volumes" Oct 30 06:30:24.119961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:24.119948 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" path="/var/lib/kubelet/pods/78c12b75-839e-427e-8cb2-9d3873ea8ee4/volumes" Oct 30 06:30:32.033379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:32.033337 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7"] Oct 30 06:30:32.044650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:32.044621 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66f3ebe5-g58k7"] Oct 30 06:30:32.118775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:32.118744 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2d840f-fe44-4329-97c8-b76dfb2ed04e" path="/var/lib/kubelet/pods/2d2d840f-fe44-4329-97c8-b76dfb2ed04e/volumes" Oct 30 06:30:33.032630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:33.032597 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg"] Oct 30 06:30:33.044359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:33.044323 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d3a2255-25zvg"] Oct 30 06:30:34.119052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:34.119017 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3912f258-783d-40cb-985a-c7c18a333e4c" path="/var/lib/kubelet/pods/3912f258-783d-40cb-985a-c7c18a333e4c/volumes" Oct 30 06:30:34.325892 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.325771 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc73e_5fcd_4a34_9465_ddbd8a86ccd6.slice/crio-5f14813ad037ed3a3879cd74a7aef53d85c60f9025ac7defc047ea5720eb3d89.scope WatchSource:0}: Error finding container 5f14813ad037ed3a3879cd74a7aef53d85c60f9025ac7defc047ea5720eb3d89: Status 404 returned error can't find the container with id 5f14813ad037ed3a3879cd74a7aef53d85c60f9025ac7defc047ea5720eb3d89 Oct 30 06:30:34.326090 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.326052 5001 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f91032_e116_4d28_bc15_a4b5144777da.slice/crio-conmon-f39727d4a98ae5c1dd2be1f46064a9e0f8b7c3395783ca9c410b1888dee73c5b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f91032_e116_4d28_bc15_a4b5144777da.slice/crio-conmon-f39727d4a98ae5c1dd2be1f46064a9e0f8b7c3395783ca9c410b1888dee73c5b.scope: no such file or directory Oct 30 06:30:34.326226 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.326189 5001 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice/crio-conmon-aceef4f1f58d2c550571185fb1a6c6cb1ffad19707fee2a13860d385af0a572a.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice/crio-conmon-aceef4f1f58d2c550571185fb1a6c6cb1ffad19707fee2a13860d385af0a572a.scope: no such file or directory Oct 30 06:30:34.326456 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.326431 5001 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice/crio-aceef4f1f58d2c550571185fb1a6c6cb1ffad19707fee2a13860d385af0a572a.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice/crio-aceef4f1f58d2c550571185fb1a6c6cb1ffad19707fee2a13860d385af0a572a.scope: no such file or directory Oct 30 06:30:34.326546 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.326469 5001 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f91032_e116_4d28_bc15_a4b5144777da.slice/crio-f39727d4a98ae5c1dd2be1f46064a9e0f8b7c3395783ca9c410b1888dee73c5b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f91032_e116_4d28_bc15_a4b5144777da.slice/crio-f39727d4a98ae5c1dd2be1f46064a9e0f8b7c3395783ca9c410b1888dee73c5b.scope: no such file or directory Oct 30 06:30:34.326546 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.326516 5001 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice/crio-conmon-9a8d0b188ace6f285c902117946eb7b59311dbaa097843a9fc0765e619ac3787.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice/crio-conmon-9a8d0b188ace6f285c902117946eb7b59311dbaa097843a9fc0765e619ac3787.scope: no such file or directory Oct 30 06:30:34.326672 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.326539 5001 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice/crio-conmon-f38d6761890f87b039f987b3b821377a79ca410e876327d7e2248f47724a424b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice/crio-conmon-f38d6761890f87b039f987b3b821377a79ca410e876327d7e2248f47724a424b.scope: no such file or directory Oct 30 06:30:34.326672 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.326581 5001 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice/crio-9a8d0b188ace6f285c902117946eb7b59311dbaa097843a9fc0765e619ac3787.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice/crio-9a8d0b188ace6f285c902117946eb7b59311dbaa097843a9fc0765e619ac3787.scope: no such file or directory Oct 30 06:30:34.326793 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.326604 5001 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice/crio-f38d6761890f87b039f987b3b821377a79ca410e876327d7e2248f47724a424b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice/crio-f38d6761890f87b039f987b3b821377a79ca410e876327d7e2248f47724a424b.scope: no such file or directory Oct 30 06:30:34.327717 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.327689 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5393b2_ac60_4342_8185_8f77a0212db0.slice/crio-6afa466f3c7af028d07525e2a8b2dff928d7759e653f5e8cf2e29753c1a778cc.scope WatchSource:0}: Error finding container 6afa466f3c7af028d07525e2a8b2dff928d7759e653f5e8cf2e29753c1a778cc: Status 404 returned error can't find the container with id 6afa466f3c7af028d07525e2a8b2dff928d7759e653f5e8cf2e29753c1a778cc Oct 30 06:30:34.328100 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.328076 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81827c5_0ba1_4a0b_b3f6_0e76ff1a2cbb.slice/crio-1b9355556df097aabd6c91e3c49730bf188b67f358ea25e3457a4066dfd12ccc.scope WatchSource:0}: Error finding container 1b9355556df097aabd6c91e3c49730bf188b67f358ea25e3457a4066dfd12ccc: Status 404 returned error can't find the container with id 1b9355556df097aabd6c91e3c49730bf188b67f358ea25e3457a4066dfd12ccc Oct 30 06:30:34.328349 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.328325 5001 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice/crio-conmon-1725b075ee92a77b184a8d7c0593a5f5754b1b307f42d086f0bf993aa463727b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice/crio-conmon-1725b075ee92a77b184a8d7c0593a5f5754b1b307f42d086f0bf993aa463727b.scope: no such file or directory Oct 30 06:30:34.328438 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.328373 5001 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice/crio-1725b075ee92a77b184a8d7c0593a5f5754b1b307f42d086f0bf993aa463727b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice/crio-1725b075ee92a77b184a8d7c0593a5f5754b1b307f42d086f0bf993aa463727b.scope: no such file or directory Oct 30 06:30:34.329238 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:34.329206 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0799a8_e504_4570_ba51_1b2aadcbfd1f.slice/crio-502be3677507fcb010c8da379383556f8bf22a792465ba96afe7be9ca3833845.scope WatchSource:0}: Error finding container 502be3677507fcb010c8da379383556f8bf22a792465ba96afe7be9ca3833845: Status 404 returned error can't find the container with id 502be3677507fcb010c8da379383556f8bf22a792465ba96afe7be9ca3833845 Oct 30 06:30:34.347734 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:34.347668 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0799a8_e504_4570_ba51_1b2aadcbfd1f.slice/crio-4c95af7c4c12621a6bc61e326bc2fdaf0bc0292cb5a55e0f584a59fe9163024f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5393b2_ac60_4342_8185_8f77a0212db0.slice/crio-7963982d9a32b2c62f1198e076a401cc6cdbd50ddc859c0cb2e9a623a8864f3c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice/crio-b6bc6c1e3c06310171e3a01ac1ca39876b09cc1e974474eb43b5a217e1988996\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f91032_e116_4d28_bc15_a4b5144777da.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81827c5_0ba1_4a0b_b3f6_0e76ff1a2cbb.slice/crio-9aa61823ec349a71f7b5222b4723f81ae099e155de0410266565320a8502f448\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice/crio-ab314e5ed407bd91d58882f479fdc643d61698b146bacc502f439e595e6a12fa\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice/crio-599c3717587b32299096e201ee530bed7e96dc856958cae2bd29f9afb90aedd2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f638843_af28_44fe_a19a_b2a04a84db0e.slice/crio-d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81827c5_0ba1_4a0b_b3f6_0e76ff1a2cbb.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc73e_5fcd_4a34_9465_ddbd8a86ccd6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0799a8_e504_4570_ba51_1b2aadcbfd1f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc73e_5fcd_4a34_9465_ddbd8a86ccd6.slice/crio-ecf8a6c5a7aef9da99d44a218d23353eb9480c22ae65c61ee7a2a50eef081b2f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice\": RecentStats: unable to find data in memory cache]" Oct 30 06:30:34.347945 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:34.347686 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-conmon-686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice/crio-dd0771f76d511ee6079e45f22950dad76c1e161c4d8dba17007dd902e914372d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0799a8_e504_4570_ba51_1b2aadcbfd1f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice/crio-ab314e5ed407bd91d58882f479fdc643d61698b146bacc502f439e595e6a12fa\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc73e_5fcd_4a34_9465_ddbd8a86ccd6.slice/crio-ecf8a6c5a7aef9da99d44a218d23353eb9480c22ae65c61ee7a2a50eef081b2f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc73e_5fcd_4a34_9465_ddbd8a86ccd6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81827c5_0ba1_4a0b_b3f6_0e76ff1a2cbb.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f638843_af28_44fe_a19a_b2a04a84db0e.slice/crio-conmon-d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f91032_e116_4d28_bc15_a4b5144777da.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0799a8_e504_4570_ba51_1b2aadcbfd1f.slice/crio-4c95af7c4c12621a6bc61e326bc2fdaf0bc0292cb5a55e0f584a59fe9163024f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice/crio-599c3717587b32299096e201ee530bed7e96dc856958cae2bd29f9afb90aedd2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-dba2d59d3b397cf2e52df92f5009258b8bf8444e61f4ae7641de9109568d7380\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81827c5_0ba1_4a0b_b3f6_0e76ff1a2cbb.slice/crio-9aa61823ec349a71f7b5222b4723f81ae099e155de0410266565320a8502f448\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5393b2_ac60_4342_8185_8f77a0212db0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice/crio-b6bc6c1e3c06310171e3a01ac1ca39876b09cc1e974474eb43b5a217e1988996\": RecentStats: unable to find data in memory cache]" Oct 30 06:30:34.347945 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:34.347705 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f91032_e116_4d28_bc15_a4b5144777da.slice/crio-cc423f27728a1d6ed2bc6ccd5a2b5e3dbeb9c2b04e807743184a71700692e19b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81827c5_0ba1_4a0b_b3f6_0e76ff1a2cbb.slice/crio-9aa61823ec349a71f7b5222b4723f81ae099e155de0410266565320a8502f448\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81827c5_0ba1_4a0b_b3f6_0e76ff1a2cbb.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-dba2d59d3b397cf2e52df92f5009258b8bf8444e61f4ae7641de9109568d7380\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc73e_5fcd_4a34_9465_ddbd8a86ccd6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f91032_e116_4d28_bc15_a4b5144777da.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f638843_af28_44fe_a19a_b2a04a84db0e.slice/crio-d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0799a8_e504_4570_ba51_1b2aadcbfd1f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice/crio-b6bc6c1e3c06310171e3a01ac1ca39876b09cc1e974474eb43b5a217e1988996\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5393b2_ac60_4342_8185_8f77a0212db0.slice/crio-7963982d9a32b2c62f1198e076a401cc6cdbd50ddc859c0cb2e9a623a8864f3c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5393b2_ac60_4342_8185_8f77a0212db0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f638843_af28_44fe_a19a_b2a04a84db0e.slice/crio-conmon-d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-conmon-686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc73e_5fcd_4a34_9465_ddbd8a86ccd6.slice/crio-ecf8a6c5a7aef9da99d44a218d23353eb9480c22ae65c61ee7a2a50eef081b2f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice/crio-dd0771f76d511ee6079e45f22950dad76c1e161c4d8dba17007dd902e914372d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0799a8_e504_4570_ba51_1b2aadcbfd1f.slice/crio-4c95af7c4c12621a6bc61e326bc2fdaf0bc0292cb5a55e0f584a59fe9163024f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice/crio-ab314e5ed407bd91d58882f479fdc643d61698b146bacc502f439e595e6a12fa\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice/crio-599c3717587b32299096e201ee530bed7e96dc856958cae2bd29f9afb90aedd2\": RecentStats: unable to find data in memory cache]" Oct 30 06:30:34.347945 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:34.347779 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f638843_af28_44fe_a19a_b2a04a84db0e.slice/crio-conmon-d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81827c5_0ba1_4a0b_b3f6_0e76ff1a2cbb.slice/crio-9aa61823ec349a71f7b5222b4723f81ae099e155de0410266565320a8502f448\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5393b2_ac60_4342_8185_8f77a0212db0.slice/crio-7963982d9a32b2c62f1198e076a401cc6cdbd50ddc859c0cb2e9a623a8864f3c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice/crio-ab314e5ed407bd91d58882f479fdc643d61698b146bacc502f439e595e6a12fa\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-conmon-686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-dba2d59d3b397cf2e52df92f5009258b8bf8444e61f4ae7641de9109568d7380\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f638843_af28_44fe_a19a_b2a04a84db0e.slice/crio-d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f91032_e116_4d28_bc15_a4b5144777da.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice/crio-dd0771f76d511ee6079e45f22950dad76c1e161c4d8dba17007dd902e914372d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0799a8_e504_4570_ba51_1b2aadcbfd1f.slice/crio-4c95af7c4c12621a6bc61e326bc2fdaf0bc0292cb5a55e0f584a59fe9163024f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5393b2_ac60_4342_8185_8f77a0212db0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc73e_5fcd_4a34_9465_ddbd8a86ccd6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice/crio-b6bc6c1e3c06310171e3a01ac1ca39876b09cc1e974474eb43b5a217e1988996\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81827c5_0ba1_4a0b_b3f6_0e76ff1a2cbb.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc73e_5fcd_4a34_9465_ddbd8a86ccd6.slice/crio-ecf8a6c5a7aef9da99d44a218d23353eb9480c22ae65c61ee7a2a50eef081b2f\": RecentStats: unable to find data in memory cache]" Oct 30 06:30:34.348231 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:34.347905 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ea84e_5daf_494f_b5e6_be8757eccb1b.slice/crio-599c3717587b32299096e201ee530bed7e96dc856958cae2bd29f9afb90aedd2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice/crio-dd0771f76d511ee6079e45f22950dad76c1e161c4d8dba17007dd902e914372d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-conmon-686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81827c5_0ba1_4a0b_b3f6_0e76ff1a2cbb.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f638843_af28_44fe_a19a_b2a04a84db0e.slice/crio-conmon-d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81827c5_0ba1_4a0b_b3f6_0e76ff1a2cbb.slice/crio-9aa61823ec349a71f7b5222b4723f81ae099e155de0410266565320a8502f448\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f91032_e116_4d28_bc15_a4b5144777da.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f638843_af28_44fe_a19a_b2a04a84db0e.slice/crio-d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a42e5_b9fc_4eff_b9da_4969663d6027.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac79889_b8fc_40c1_a988_946e1525dd2a.slice/crio-b6bc6c1e3c06310171e3a01ac1ca39876b09cc1e974474eb43b5a217e1988996\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c12b75_839e_427e_8cb2_9d3873ea8ee4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0799a8_e504_4570_ba51_1b2aadcbfd1f.slice/crio-4c95af7c4c12621a6bc61e326bc2fdaf0bc0292cb5a55e0f584a59fe9163024f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5393b2_ac60_4342_8185_8f77a0212db0.slice/crio-7963982d9a32b2c62f1198e076a401cc6cdbd50ddc859c0cb2e9a623a8864f3c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5393b2_ac60_4342_8185_8f77a0212db0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccfc73e_5fcd_4a34_9465_ddbd8a86ccd6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18bb796_a33f_41bc_93f2_302c700fd101.slice/crio-686eaeb0548ffbe4aeb3a7a777e3c36b4926f9c52df8530b5a64ceaa73ef6927.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0799a8_e504_4570_ba51_1b2aadcbfd1f.slice\": RecentStats: unable to find data in memory cache]" Oct 30 06:30:34.539685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:34.539661 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" Oct 30 06:30:34.661132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:34.660409 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f638843-af28-44fe-a19a-b2a04a84db0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f638843-af28-44fe-a19a-b2a04a84db0e-maas-api-service-ca\") pod \"4f638843-af28-44fe-a19a-b2a04a84db0e\" (UID: \"4f638843-af28-44fe-a19a-b2a04a84db0e\") " Oct 30 06:30:34.661132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:34.660614 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f638843-af28-44fe-a19a-b2a04a84db0e-kube-api-access-2jjr2\" (UniqueName: \"kubernetes.io/projected/4f638843-af28-44fe-a19a-b2a04a84db0e-kube-api-access-2jjr2\") pod \"4f638843-af28-44fe-a19a-b2a04a84db0e\" (UID: \"4f638843-af28-44fe-a19a-b2a04a84db0e\") " Oct 30 06:30:34.661843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:34.661808 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f638843-af28-44fe-a19a-b2a04a84db0e-maas-api-service-ca" pod "4f638843-af28-44fe-a19a-b2a04a84db0e" (UID: "4f638843-af28-44fe-a19a-b2a04a84db0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:34.663875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:34.663848 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f638843-af28-44fe-a19a-b2a04a84db0e-kube-api-access-2jjr2" pod "4f638843-af28-44fe-a19a-b2a04a84db0e" (UID: "4f638843-af28-44fe-a19a-b2a04a84db0e"). InnerVolumeSpecName "kube-api-access-2jjr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:34.763046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:34.763011 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f638843-af28-44fe-a19a-b2a04a84db0e-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:34.763046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:34.763044 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jjr2\" (UniqueName: \"kubernetes.io/projected/4f638843-af28-44fe-a19a-b2a04a84db0e-kube-api-access-2jjr2\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:35.067017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.066918 5001 generic.go:356] "Generic (PLEG): container finished" podID="4f638843-af28-44fe-a19a-b2a04a84db0e" containerID="d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e" exitCode=137 Oct 30 06:30:35.067017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.066969 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" Oct 30 06:30:35.067277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.067002 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" event={"ID":"4f638843-af28-44fe-a19a-b2a04a84db0e","Type":"ContainerDied","Data":"d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e"} Oct 30 06:30:35.067277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.067093 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv" event={"ID":"4f638843-af28-44fe-a19a-b2a04a84db0e","Type":"ContainerDied","Data":"86ade3abfe0e486bcc49636f43908e6c5f88ae9a4385380fc7250b9235981d4f"} Oct 30 06:30:35.067277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.067124 5001 scope.go:122] "RemoveContainer" containerID="d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e" Oct 30 06:30:35.079348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.079322 5001 scope.go:122] "RemoveContainer" containerID="b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b" Oct 30 06:30:35.091515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.091491 5001 scope.go:122] "RemoveContainer" containerID="d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e" Oct 30 06:30:35.091815 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:35.091791 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e\": container with ID starting with d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e not found: ID does not exist" containerID="d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e" Oct 30 06:30:35.091949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.091821 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e"} err="failed to get container status \"d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e\": rpc error: code = NotFound desc = could not find container \"d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e\": container with ID starting with d81a84c5db782b04246d2fd8773dd9ce263584e90b27bc317f6a826b0d84ad2e not found: ID does not exist" Oct 30 06:30:35.091949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.091840 5001 scope.go:122] "RemoveContainer" containerID="b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b" Oct 30 06:30:35.092114 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:35.092095 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b\": container with ID starting with b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b not found: ID does not exist" containerID="b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b" Oct 30 06:30:35.092176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.092121 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b"} err="failed to get container status \"b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b\": rpc error: code = NotFound desc = could not find container \"b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b\": container with ID starting with b0c9dd9b29d714506144c906dc01136ce440870b8f736493ca782a77af63926b not found: ID does not exist" Oct 30 06:30:35.118887 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.118854 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv"] Oct 30 06:30:35.131112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:35.131087 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-3bbfc8-29889030-h84hv"] Oct 30 06:30:36.119076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:36.119036 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f638843-af28-44fe-a19a-b2a04a84db0e" path="/var/lib/kubelet/pods/4f638843-af28-44fe-a19a-b2a04a84db0e/volumes" Oct 30 06:30:38.272487 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:38.272448 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:30:40.498328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:40.492896 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b"] Oct 30 06:30:40.498328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:40.493222 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" podUID="69144896-68b9-4b57-9e4c-8bca3f96c640" containerName="istio-proxy" containerID="cri-o://1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1" gracePeriod=30 Oct 30 06:30:45.247348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.247290 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:30:45.451243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.451195 5001 patch_prober.go:29] interesting pod/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:30:45.451472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.451256 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" podUID="69144896-68b9-4b57-9e4c-8bca3f96c640" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:30:45.835290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.835260 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:45.937075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.937033 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-token\" (UniqueName: \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-token\") pod \"69144896-68b9-4b57-9e4c-8bca3f96c640\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " Oct 30 06:30:45.937291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.937145 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-envoy\") pod \"69144896-68b9-4b57-9e4c-8bca3f96c640\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " Oct 30 06:30:45.937291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.937193 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-certs\") pod \"69144896-68b9-4b57-9e4c-8bca3f96c640\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " Oct 30 06:30:45.937291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.937242 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-data\") pod \"69144896-68b9-4b57-9e4c-8bca3f96c640\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " Oct 30 06:30:45.937291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.937281 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-credential-socket\") pod \"69144896-68b9-4b57-9e4c-8bca3f96c640\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " Oct 30 06:30:45.937697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.937605 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-credential-socket" pod "69144896-68b9-4b57-9e4c-8bca3f96c640" (UID: "69144896-68b9-4b57-9e4c-8bca3f96c640"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:45.938614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.938567 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-certs" pod "69144896-68b9-4b57-9e4c-8bca3f96c640" (UID: "69144896-68b9-4b57-9e4c-8bca3f96c640"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:45.938820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.938612 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-data" pod "69144896-68b9-4b57-9e4c-8bca3f96c640" (UID: "69144896-68b9-4b57-9e4c-8bca3f96c640"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.943653 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-podinfo\") pod \"69144896-68b9-4b57-9e4c-8bca3f96c640\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.943694 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-envoy" pod "69144896-68b9-4b57-9e4c-8bca3f96c640" (UID: "69144896-68b9-4b57-9e4c-8bca3f96c640"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.943721 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69144896-68b9-4b57-9e4c-8bca3f96c640-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69144896-68b9-4b57-9e4c-8bca3f96c640-istiod-ca-cert\") pod \"69144896-68b9-4b57-9e4c-8bca3f96c640\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.943792 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-socket\") pod \"69144896-68b9-4b57-9e4c-8bca3f96c640\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.943847 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-token" pod "69144896-68b9-4b57-9e4c-8bca3f96c640" (UID: "69144896-68b9-4b57-9e4c-8bca3f96c640"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.943887 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-kube-api-access-dmkgp\" (UniqueName: \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-kube-api-access-dmkgp\") pod \"69144896-68b9-4b57-9e4c-8bca3f96c640\" (UID: \"69144896-68b9-4b57-9e4c-8bca3f96c640\") " Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.944102 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-socket" pod "69144896-68b9-4b57-9e4c-8bca3f96c640" (UID: "69144896-68b9-4b57-9e4c-8bca3f96c640"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.944145 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69144896-68b9-4b57-9e4c-8bca3f96c640-istiod-ca-cert" pod "69144896-68b9-4b57-9e4c-8bca3f96c640" (UID: "69144896-68b9-4b57-9e4c-8bca3f96c640"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.944703 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69144896-68b9-4b57-9e4c-8bca3f96c640-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.944724 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.944739 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.944752 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.944765 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.944777 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:45.944916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.944790 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69144896-68b9-4b57-9e4c-8bca3f96c640-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:45.946649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.946623 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-podinfo" pod "69144896-68b9-4b57-9e4c-8bca3f96c640" (UID: "69144896-68b9-4b57-9e4c-8bca3f96c640"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:30:45.946790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:45.946772 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-kube-api-access-dmkgp" pod "69144896-68b9-4b57-9e4c-8bca3f96c640" (UID: "69144896-68b9-4b57-9e4c-8bca3f96c640"). InnerVolumeSpecName "kube-api-access-dmkgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:46.041332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.041227 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx"] Oct 30 06:30:46.046171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.046140 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmkgp\" (UniqueName: \"kubernetes.io/projected/69144896-68b9-4b57-9e4c-8bca3f96c640-kube-api-access-dmkgp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:46.046171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.046172 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69144896-68b9-4b57-9e4c-8bca3f96c640-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:46.047041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.047016 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805327ee-57gzx"] Oct 30 06:30:46.121080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.121048 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a35c97-31ea-4ed3-b993-17047dc03835" path="/var/lib/kubelet/pods/80a35c97-31ea-4ed3-b993-17047dc03835/volumes" Oct 30 06:30:46.187086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.187054 5001 generic.go:356] "Generic (PLEG): container finished" podID="69144896-68b9-4b57-9e4c-8bca3f96c640" containerID="1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1" exitCode=0 Oct 30 06:30:46.187286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.187182 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" Oct 30 06:30:46.187286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.187210 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" event={"ID":"69144896-68b9-4b57-9e4c-8bca3f96c640","Type":"ContainerDied","Data":"1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1"} Oct 30 06:30:46.187286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.187278 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b" event={"ID":"69144896-68b9-4b57-9e4c-8bca3f96c640","Type":"ContainerDied","Data":"9632b9ec20c7e28ad7bb3c17a6b571a6a76b7614b5a24e92a36241bccf2c6b95"} Oct 30 06:30:46.187462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.187349 5001 scope.go:122] "RemoveContainer" containerID="1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1" Oct 30 06:30:46.200601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.200581 5001 scope.go:122] "RemoveContainer" containerID="1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1" Oct 30 06:30:46.200922 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:30:46.200903 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1\": container with ID starting with 1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1 not found: ID does not exist" containerID="1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1" Oct 30 06:30:46.201036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.200930 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1"} err="failed to get container status \"1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1\": rpc error: code = NotFound desc = could not find container \"1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1\": container with ID starting with 1d6674f9e8f7f4930f87bcd206b127065ae0fed99336f2ad7e96424add3958e1 not found: ID does not exist" Oct 30 06:30:46.243264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.243191 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b"] Oct 30 06:30:46.247871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:46.247844 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-b8b590-openshift-default-9bdfcd69-hzk2b"] Oct 30 06:30:48.119689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:48.119654 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69144896-68b9-4b57-9e4c-8bca3f96c640" path="/var/lib/kubelet/pods/69144896-68b9-4b57-9e4c-8bca3f96c640/volumes" Oct 30 06:30:49.004790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.004751 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9"] Oct 30 06:30:49.005515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005500 5001 cpu_manager.go:414] "containerMap: removing container" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005517 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005527 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005532 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005540 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005546 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005557 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005563 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005574 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005578 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005585 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005589 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerName="cleanup" Oct 30 06:30:49.005593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005597 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005602 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005612 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005616 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005626 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4f638843-af28-44fe-a19a-b2a04a84db0e" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005631 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f638843-af28-44fe-a19a-b2a04a84db0e" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005637 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005641 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005646 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005650 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005659 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005663 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005668 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005672 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005678 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005683 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005691 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005696 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005709 5001 cpu_manager.go:414] "containerMap: removing container" podUID="69144896-68b9-4b57-9e4c-8bca3f96c640" containerName="istio-proxy" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005713 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69144896-68b9-4b57-9e4c-8bca3f96c640" containerName="istio-proxy" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005729 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005733 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005738 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005742 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005748 5001 cpu_manager.go:414] "containerMap: removing container" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005752 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005758 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005762 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005769 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005774 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005780 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005784 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005790 5001 cpu_manager.go:414] "containerMap: removing container" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005794 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005897 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005904 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005910 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005915 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005921 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005925 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005931 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005935 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005940 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005945 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005950 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f638843-af28-44fe-a19a-b2a04a84db0e" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005954 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005959 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005964 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c12b75-839e-427e-8cb2-9d3873ea8ee4" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005969 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005975 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005981 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005986 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005993 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.005999 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerName="cleanup" Oct 30 06:30:49.005978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.006004 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerName="cleanup" Oct 30 06:30:49.007681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.006010 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81827c5-0ba1-4a0b-b3f6-0e76ff1a2cbb" containerName="cleanup" Oct 30 06:30:49.007681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.006016 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerName="cleanup" Oct 30 06:30:49.007681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.006021 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerName="cleanup" Oct 30 06:30:49.007681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.006026 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerName="cleanup" Oct 30 06:30:49.007681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.006031 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="69144896-68b9-4b57-9e4c-8bca3f96c640" containerName="istio-proxy" Oct 30 06:30:49.011237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.011216 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:49.027015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.026988 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:30:49.029867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.029841 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9"] Oct 30 06:30:49.034909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.034881 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:30:49.038872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.038846 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:30:49.039034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.038998 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:30:49.042547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.042520 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29"] Oct 30 06:30:49.049560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.049530 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0651b4ba-52z29"] Oct 30 06:30:49.116348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.116300 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxnlt\" (UniqueName: \"kubernetes.io/projected/f321b71b-f01f-40fd-b934-05bece520a17-kube-api-access-jxnlt\") pod \"maas-api-revoke-sub-98ecfb0f-kx9f9\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:49.116514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.116447 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ecfb0f-kx9f9\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:49.116514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.116476 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ecfb0f-kx9f9\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:49.217757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.217714 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ecfb0f-kx9f9\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:49.218289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.217769 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ecfb0f-kx9f9\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:49.218289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.217966 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxnlt\" (UniqueName: \"kubernetes.io/projected/f321b71b-f01f-40fd-b934-05bece520a17-kube-api-access-jxnlt\") pod \"maas-api-revoke-sub-98ecfb0f-kx9f9\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:49.218582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.218515 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ecfb0f-kx9f9\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:49.220536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.220514 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ecfb0f-kx9f9\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:49.241390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.241286 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxnlt\" (UniqueName: \"kubernetes.io/projected/f321b71b-f01f-40fd-b934-05bece520a17-kube-api-access-jxnlt\") pod \"maas-api-revoke-sub-98ecfb0f-kx9f9\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:49.326352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.326257 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:49.642202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:49.642162 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9"] Oct 30 06:30:50.120901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:50.120867 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a3762c-7164-4716-8ab6-a24fe8c6a22b" path="/var/lib/kubelet/pods/49a3762c-7164-4716-8ab6-a24fe8c6a22b/volumes" Oct 30 06:30:50.231280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:50.231193 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" event={"ID":"f321b71b-f01f-40fd-b934-05bece520a17","Type":"ContainerStarted","Data":"a37221e0c28844fddc124bc72338798ddb99b6a517ca6b1b3a715b0a0c2ed6c8"} Oct 30 06:30:50.231280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:50.231256 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" event={"ID":"f321b71b-f01f-40fd-b934-05bece520a17","Type":"ContainerStarted","Data":"85af572e40302ebdf06ee5538e127149eb610093084186be54a5fb3df228ec12"} Oct 30 06:30:50.273036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:50.272989 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" podStartSLOduration=2.272975742 podStartE2EDuration="2.272975742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:50.269500841 +0000 UTC m=+2264.265457850" watchObservedRunningTime="2026-10-30 06:30:50.272975742 +0000 UTC m=+2264.268932752" Oct 30 06:30:51.241823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:51.241790 5001 generic.go:356] "Generic (PLEG): container finished" podID="f321b71b-f01f-40fd-b934-05bece520a17" containerID="a37221e0c28844fddc124bc72338798ddb99b6a517ca6b1b3a715b0a0c2ed6c8" exitCode=0 Oct 30 06:30:51.242230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:51.241877 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" event={"ID":"f321b71b-f01f-40fd-b934-05bece520a17","Type":"ContainerDied","Data":"a37221e0c28844fddc124bc72338798ddb99b6a517ca6b1b3a715b0a0c2ed6c8"} Oct 30 06:30:52.036271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.036233 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4"] Oct 30 06:30:52.047614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.047585 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af895423-9ncg4"] Oct 30 06:30:52.121190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.121149 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a45917-f6c7-4b51-b519-113448b07bb9" path="/var/lib/kubelet/pods/35a45917-f6c7-4b51-b519-113448b07bb9/volumes" Oct 30 06:30:52.439839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.439812 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:52.481343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.481280 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f321b71b-f01f-40fd-b934-05bece520a17-kube-api-access-jxnlt\" (UniqueName: \"kubernetes.io/projected/f321b71b-f01f-40fd-b934-05bece520a17-kube-api-access-jxnlt\") pod \"f321b71b-f01f-40fd-b934-05bece520a17\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " Oct 30 06:30:52.484180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.481684 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-openshift-ca\") pod \"f321b71b-f01f-40fd-b934-05bece520a17\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " Oct 30 06:30:52.484180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.482147 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-xks-ca\") pod \"f321b71b-f01f-40fd-b934-05bece520a17\" (UID: \"f321b71b-f01f-40fd-b934-05bece520a17\") " Oct 30 06:30:52.484180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.482073 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-openshift-ca" pod "f321b71b-f01f-40fd-b934-05bece520a17" (UID: "f321b71b-f01f-40fd-b934-05bece520a17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:52.484180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.482982 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:52.484180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.484124 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f321b71b-f01f-40fd-b934-05bece520a17-kube-api-access-jxnlt" pod "f321b71b-f01f-40fd-b934-05bece520a17" (UID: "f321b71b-f01f-40fd-b934-05bece520a17"). InnerVolumeSpecName "kube-api-access-jxnlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:52.485825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.485798 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-xks-ca" pod "f321b71b-f01f-40fd-b934-05bece520a17" (UID: "f321b71b-f01f-40fd-b934-05bece520a17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:30:52.584056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.583968 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxnlt\" (UniqueName: \"kubernetes.io/projected/f321b71b-f01f-40fd-b934-05bece520a17-kube-api-access-jxnlt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:52.584056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:52.583995 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f321b71b-f01f-40fd-b934-05bece520a17-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:30:53.265636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:53.265602 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" Oct 30 06:30:53.265819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:53.265641 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9" event={"ID":"f321b71b-f01f-40fd-b934-05bece520a17","Type":"ContainerDied","Data":"85af572e40302ebdf06ee5538e127149eb610093084186be54a5fb3df228ec12"} Oct 30 06:30:53.265819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:53.265695 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85af572e40302ebdf06ee5538e127149eb610093084186be54a5fb3df228ec12" Oct 30 06:30:59.354246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.354209 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl"] Oct 30 06:30:59.354938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.354920 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerName="cleanup" Oct 30 06:30:59.354938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.354939 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.354957 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f321b71b-f01f-40fd-b934-05bece520a17" containerName="revoke-keys" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.354963 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f321b71b-f01f-40fd-b934-05bece520a17" containerName="revoke-keys" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.354970 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.354974 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5393b2-ac60-4342-8185-8f77a0212db0" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.354986 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.354991 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355012 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355017 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355036 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355040 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355050 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355055 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355065 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerName="cleanup" Oct 30 06:30:59.355065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355070 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0799a8-e504-4570-ba51-1b2aadcbfd1f" containerName="cleanup" Oct 30 06:30:59.355837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355079 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" Oct 30 06:30:59.355837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355083 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18bb796-a33f-41bc-93f2-302c700fd101" containerName="istio-proxy" Oct 30 06:30:59.355837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355092 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4f638843-af28-44fe-a19a-b2a04a84db0e" containerName="cleanup" Oct 30 06:30:59.355837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355096 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f638843-af28-44fe-a19a-b2a04a84db0e" containerName="cleanup" Oct 30 06:30:59.355837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355186 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f321b71b-f01f-40fd-b934-05bece520a17" containerName="revoke-keys" Oct 30 06:30:59.355837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355196 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ccfc73e-5fcd-4a34-9465-ddbd8a86ccd6" containerName="cleanup" Oct 30 06:30:59.355837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355204 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac79889-b8fc-40c1-a988-946e1525dd2a" containerName="cleanup" Oct 30 06:30:59.355837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355209 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0a42e5-b9fc-4eff-b9da-4969663d6027" containerName="cleanup" Oct 30 06:30:59.355837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355216 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f638843-af28-44fe-a19a-b2a04a84db0e" containerName="cleanup" Oct 30 06:30:59.355837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355222 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f91032-e116-4d28-bc15-a4b5144777da" containerName="cleanup" Oct 30 06:30:59.355837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.355230 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0ea84e-5daf-494f-b5e6-be8757eccb1b" containerName="cleanup" Oct 30 06:30:59.358731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.358709 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:30:59.373661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.373638 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:30:59.381439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.381409 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl"] Oct 30 06:30:59.385428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.385296 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:30:59.385428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.385391 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:30:59.385617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.385471 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:30:59.511156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.511116 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjn6g\" (UniqueName: \"kubernetes.io/projected/884a0db4-39db-4613-a5fb-6fc464035d0e-kube-api-access-wjn6g\") pod \"maas-api-revoke-sub-60d8da6f-xvwfl\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:30:59.511393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.511270 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d8da6f-xvwfl\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:30:59.511456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.511425 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d8da6f-xvwfl\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:30:59.612813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.612780 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjn6g\" (UniqueName: \"kubernetes.io/projected/884a0db4-39db-4613-a5fb-6fc464035d0e-kube-api-access-wjn6g\") pod \"maas-api-revoke-sub-60d8da6f-xvwfl\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:30:59.613017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.612878 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d8da6f-xvwfl\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:30:59.613017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.612920 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d8da6f-xvwfl\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:30:59.613656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.613634 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d8da6f-xvwfl\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:30:59.615431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.615407 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d8da6f-xvwfl\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:30:59.635535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.635510 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjn6g\" (UniqueName: \"kubernetes.io/projected/884a0db4-39db-4613-a5fb-6fc464035d0e-kube-api-access-wjn6g\") pod \"maas-api-revoke-sub-60d8da6f-xvwfl\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:30:59.674204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.674164 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:30:59.993963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:30:59.993877 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl"] Oct 30 06:30:59.998344 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:30:59.998263 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884a0db4_39db_4613_a5fb_6fc464035d0e.slice/crio-c028bb631faa1a02379e085abcf9041f382d6335a7f8f62f4c037553638badde WatchSource:0}: Error finding container c028bb631faa1a02379e085abcf9041f382d6335a7f8f62f4c037553638badde: Status 404 returned error can't find the container with id c028bb631faa1a02379e085abcf9041f382d6335a7f8f62f4c037553638badde Oct 30 06:31:00.334410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:00.334330 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" event={"ID":"884a0db4-39db-4613-a5fb-6fc464035d0e","Type":"ContainerStarted","Data":"c028bb631faa1a02379e085abcf9041f382d6335a7f8f62f4c037553638badde"} Oct 30 06:31:01.348146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:01.348059 5001 generic.go:356] "Generic (PLEG): container finished" podID="884a0db4-39db-4613-a5fb-6fc464035d0e" containerID="02e56a8ef754a00af780c63ec18483aed1010bb5b1a9fc0db76801b289d95bea" exitCode=0 Oct 30 06:31:01.348146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:01.348134 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" event={"ID":"884a0db4-39db-4613-a5fb-6fc464035d0e","Type":"ContainerDied","Data":"02e56a8ef754a00af780c63ec18483aed1010bb5b1a9fc0db76801b289d95bea"} Oct 30 06:31:02.033549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.033513 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv"] Oct 30 06:31:02.059532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.059492 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842c8d2b-xxxfv"] Oct 30 06:31:02.139408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.139371 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b085c0c-26dd-40b6-b038-49483722314c" path="/var/lib/kubelet/pods/2b085c0c-26dd-40b6-b038-49483722314c/volumes" Oct 30 06:31:02.529465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.529442 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:31:02.664324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.664214 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-openshift-ca\") pod \"884a0db4-39db-4613-a5fb-6fc464035d0e\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " Oct 30 06:31:02.664496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.664404 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-xks-ca\") pod \"884a0db4-39db-4613-a5fb-6fc464035d0e\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " Oct 30 06:31:02.664578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.664536 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884a0db4-39db-4613-a5fb-6fc464035d0e-kube-api-access-wjn6g\" (UniqueName: \"kubernetes.io/projected/884a0db4-39db-4613-a5fb-6fc464035d0e-kube-api-access-wjn6g\") pod \"884a0db4-39db-4613-a5fb-6fc464035d0e\" (UID: \"884a0db4-39db-4613-a5fb-6fc464035d0e\") " Oct 30 06:31:02.664709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.664677 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-openshift-ca" pod "884a0db4-39db-4613-a5fb-6fc464035d0e" (UID: "884a0db4-39db-4613-a5fb-6fc464035d0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:02.665428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.665391 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:02.667110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.667079 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884a0db4-39db-4613-a5fb-6fc464035d0e-kube-api-access-wjn6g" pod "884a0db4-39db-4613-a5fb-6fc464035d0e" (UID: "884a0db4-39db-4613-a5fb-6fc464035d0e"). InnerVolumeSpecName "kube-api-access-wjn6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:02.667231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.667192 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-xks-ca" pod "884a0db4-39db-4613-a5fb-6fc464035d0e" (UID: "884a0db4-39db-4613-a5fb-6fc464035d0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:31:02.766512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.766470 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjn6g\" (UniqueName: \"kubernetes.io/projected/884a0db4-39db-4613-a5fb-6fc464035d0e-kube-api-access-wjn6g\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:02.766512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:02.766507 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884a0db4-39db-4613-a5fb-6fc464035d0e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:03.373434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:03.373399 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" event={"ID":"884a0db4-39db-4613-a5fb-6fc464035d0e","Type":"ContainerDied","Data":"c028bb631faa1a02379e085abcf9041f382d6335a7f8f62f4c037553638badde"} Oct 30 06:31:03.373619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:03.373457 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c028bb631faa1a02379e085abcf9041f382d6335a7f8f62f4c037553638badde" Oct 30 06:31:03.373619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:03.373415 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl" Oct 30 06:31:04.085506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:04.085458 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr"] Oct 30 06:31:04.095982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:04.095937 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6c9c4e-99pcr"] Oct 30 06:31:04.121597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:04.121560 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45867f92-1f2e-4c09-9c71-3ed44574a01d" path="/var/lib/kubelet/pods/45867f92-1f2e-4c09-9c71-3ed44574a01d/volumes" Oct 30 06:31:08.270365 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:31:08.270274 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:31:09.410672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:09.410635 5001 scope.go:122] "RemoveContainer" containerID="4cf06900233868f5640019138ad91c604c13f8c132dbe0b6e68c6ea25f60632c" Oct 30 06:31:09.421868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:09.421844 5001 scope.go:122] "RemoveContainer" containerID="7027c571628cbe6be536982e6525334f026ff78c0c99a4c9582175f9fbe14c5f" Oct 30 06:31:09.433971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:09.433947 5001 scope.go:122] "RemoveContainer" containerID="e3e2e23db723f7ef724a8ddc2042a8c8659a250685a27759daab5380621a3db9" Oct 30 06:31:09.446109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:09.446078 5001 scope.go:122] "RemoveContainer" containerID="e1259295f3855d23e0ae090cda4cf146b64e1ad1bc0ee87160c17684e86294dc" Oct 30 06:31:09.458948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:09.458925 5001 scope.go:122] "RemoveContainer" containerID="8df8a01e78de008ba032dcf64faccd071925dc3e858f4e432b711168992cf72b" Oct 30 06:31:09.472402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:09.472372 5001 scope.go:122] "RemoveContainer" containerID="c76bc7ad2e5817141da9f96490d1ebf32a5458a397c40683015acc9d699de800" Oct 30 06:31:09.485703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:09.485678 5001 scope.go:122] "RemoveContainer" containerID="936f936bd93019790b694c558ec63e204060fbd18d259b9fd38a0a302a88b451" Oct 30 06:31:09.496813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:09.496787 5001 scope.go:122] "RemoveContainer" containerID="9b3931b2ceb53a629bf1d7a143a7a5f1c060c6fa9d724422636355e49747f79a" Oct 30 06:31:09.507114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:09.507089 5001 scope.go:122] "RemoveContainer" containerID="355c57cad80142931844c1dec499a106f34f32f33c8d7f07ee0ed5edff0b1d58" Oct 30 06:31:09.516963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:09.516943 5001 scope.go:122] "RemoveContainer" containerID="1b2d483c08b28b0292337975c816af6b9c3d93b2df67f7cd58fa9b5c50a75178" Oct 30 06:31:09.527256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:09.527238 5001 scope.go:122] "RemoveContainer" containerID="cf62ea56c177222b92ddfae09c62a70724692e59ff7d18711e1a13847f065a86" Oct 30 06:31:13.036081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:13.036043 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7"] Oct 30 06:31:13.047103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:13.047077 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36eb8991-sn2g7"] Oct 30 06:31:14.121631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:14.121594 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b76b59-27df-4697-9670-de78c1cc0c08" path="/var/lib/kubelet/pods/22b76b59-27df-4697-9670-de78c1cc0c08/volumes" Oct 30 06:31:19.033855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:19.033768 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22"] Oct 30 06:31:19.045255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:19.045224 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50ddb1c-cwj22"] Oct 30 06:31:20.119395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:20.119357 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99cf375-73be-437b-8317-399867eea0ac" path="/var/lib/kubelet/pods/e99cf375-73be-437b-8317-399867eea0ac/volumes" Oct 30 06:31:24.435500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.435460 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7324-6ab4a8"] Oct 30 06:31:24.436245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.436228 5001 cpu_manager.go:414] "containerMap: removing container" podUID="884a0db4-39db-4613-a5fb-6fc464035d0e" containerName="revoke-keys" Oct 30 06:31:24.436392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.436247 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884a0db4-39db-4613-a5fb-6fc464035d0e" containerName="revoke-keys" Oct 30 06:31:24.436452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.436439 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="884a0db4-39db-4613-a5fb-6fc464035d0e" containerName="revoke-keys" Oct 30 06:31:24.443601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.443579 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7324-6ab4a8" Oct 30 06:31:24.454631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.454605 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7324-6ab4a8"] Oct 30 06:31:24.457786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.457755 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p6zmk\"" Oct 30 06:31:24.457786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.457778 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 06:31:24.457953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.457809 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 06:31:24.545223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.545181 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhb5v\" (UniqueName: \"kubernetes.io/projected/e2c4d6f2-ccde-4c36-a612-9f26e21947e1-kube-api-access-nhb5v\") pod \"test-curl-7324-6ab4a8\" (UID: \"e2c4d6f2-ccde-4c36-a612-9f26e21947e1\") " pod="opendatahub/test-curl-7324-6ab4a8" Oct 30 06:31:24.646464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.646425 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhb5v\" (UniqueName: \"kubernetes.io/projected/e2c4d6f2-ccde-4c36-a612-9f26e21947e1-kube-api-access-nhb5v\") pod \"test-curl-7324-6ab4a8\" (UID: \"e2c4d6f2-ccde-4c36-a612-9f26e21947e1\") " pod="opendatahub/test-curl-7324-6ab4a8" Oct 30 06:31:24.670278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.670240 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhb5v\" (UniqueName: \"kubernetes.io/projected/e2c4d6f2-ccde-4c36-a612-9f26e21947e1-kube-api-access-nhb5v\") pod \"test-curl-7324-6ab4a8\" (UID: \"e2c4d6f2-ccde-4c36-a612-9f26e21947e1\") " pod="opendatahub/test-curl-7324-6ab4a8" Oct 30 06:31:24.758763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:24.758672 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7324-6ab4a8" Oct 30 06:31:25.112800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:25.111724 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7324-6ab4a8"] Oct 30 06:31:25.125004 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:31:25.124957 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2c4d6f2_ccde_4c36_a612_9f26e21947e1.slice/crio-7728f25f2f158fe1892665820d361ac76dd3a5a0ca28d41b6aba01aa41646eaa WatchSource:0}: Error finding container 7728f25f2f158fe1892665820d361ac76dd3a5a0ca28d41b6aba01aa41646eaa: Status 404 returned error can't find the container with id 7728f25f2f158fe1892665820d361ac76dd3a5a0ca28d41b6aba01aa41646eaa Oct 30 06:31:25.134036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:25.133395 5001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:31:25.596605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:25.596524 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7324-6ab4a8" event={"ID":"e2c4d6f2-ccde-4c36-a612-9f26e21947e1","Type":"ContainerStarted","Data":"7728f25f2f158fe1892665820d361ac76dd3a5a0ca28d41b6aba01aa41646eaa"} Oct 30 06:31:28.679823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:28.679768 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7324-6ab4a8" event={"ID":"e2c4d6f2-ccde-4c36-a612-9f26e21947e1","Type":"ContainerStarted","Data":"7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d"} Oct 30 06:31:28.724883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:28.724827 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7324-6ab4a8" podStartSLOduration=2.076363206 podStartE2EDuration="4.724811108s" totalImagesPullingTime="2.648447902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:28.722347561 +0000 UTC m=+2302.718304565" watchObservedRunningTime="2026-10-30 06:31:28.724811108 +0000 UTC m=+2302.720768119" Oct 30 06:31:29.664595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:29.664559 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7324-6ab4a8"] Oct 30 06:31:29.671721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:29.671686 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7324-6ab4a8"] Oct 30 06:31:30.031791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:30.031660 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj"] Oct 30 06:31:30.043077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:30.043045 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257bae11-dxjnj"] Oct 30 06:31:30.118409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:30.118378 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc245d4-e56a-4578-b3d5-ea90cb95388a" path="/var/lib/kubelet/pods/3bc245d4-e56a-4578-b3d5-ea90cb95388a/volumes" Oct 30 06:31:30.702859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:30.702789 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7324-6ab4a8" podUID="e2c4d6f2-ccde-4c36-a612-9f26e21947e1" containerName="test-curl-7324-6ab4a8" containerID="cri-o://7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d" gracePeriod=2 Oct 30 06:31:31.787174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.787137 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6"] Oct 30 06:31:31.788273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.788244 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e2c4d6f2-ccde-4c36-a612-9f26e21947e1" containerName="test-curl-7324-6ab4a8" Oct 30 06:31:31.788428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.788273 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c4d6f2-ccde-4c36-a612-9f26e21947e1" containerName="test-curl-7324-6ab4a8" Oct 30 06:31:31.788637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.788618 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c4d6f2-ccde-4c36-a612-9f26e21947e1" containerName="test-curl-7324-6ab4a8" Oct 30 06:31:31.793001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.792975 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:31.814602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.814565 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2d2923-openshift-default-dockercfg-pmrkx\"" Oct 30 06:31:31.828847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.828812 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6"] Oct 30 06:31:31.972329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.971429 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-envoy\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:31.972329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.971503 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-token\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:31.972329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.971565 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvbw\" (UniqueName: \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-kube-api-access-wcvbw\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:31.972329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.971599 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-credential-socket\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:31.972329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.971679 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-socket\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:31.972329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.971755 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ca4d25-1038-4aaa-9d74-436b3222223c-istiod-ca-cert\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:31.972329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.971832 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-data\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:31.972329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.971862 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-podinfo\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:31.972329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:31.971938 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-certs\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.074220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.074121 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-data\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.074496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.074474 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-podinfo\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.074650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.074637 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-certs\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.074836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.074821 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-envoy\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.074944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.074931 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-token\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.075086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.075052 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvbw\" (UniqueName: \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-kube-api-access-wcvbw\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.075192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.075181 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-credential-socket\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.075361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.075348 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-socket\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.075879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.075855 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ca4d25-1038-4aaa-9d74-436b3222223c-istiod-ca-cert\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.076913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.076889 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ca4d25-1038-4aaa-9d74-436b3222223c-istiod-ca-cert\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.080687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.080664 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-data\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.102534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.102496 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-certs\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.103966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.103029 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-socket\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.104138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.103908 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-credential-socket\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.130343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.124119 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-envoy\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.162446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.162413 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvbw\" (UniqueName: \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-kube-api-access-wcvbw\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.163119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.163045 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-token\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.174325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.170950 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-podinfo\") pod \"e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:32.433622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:32.433563 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:33.042691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.042664 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7324-6ab4a8" Oct 30 06:31:33.082338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.082267 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6"] Oct 30 06:31:33.084706 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:31:33.084673 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ca4d25_1038_4aaa_9d74_436b3222223c.slice/crio-7d4a9e26f0329c4acbda9aa2ee66f551797dcbfd469875ca49dcf0ac318e7657 WatchSource:0}: Error finding container 7d4a9e26f0329c4acbda9aa2ee66f551797dcbfd469875ca49dcf0ac318e7657: Status 404 returned error can't find the container with id 7d4a9e26f0329c4acbda9aa2ee66f551797dcbfd469875ca49dcf0ac318e7657 Oct 30 06:31:33.087849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.087807 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:31:33.087959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.087911 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:31:33.088002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.087959 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:31:33.197999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.197967 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c4d6f2-ccde-4c36-a612-9f26e21947e1-kube-api-access-nhb5v\" (UniqueName: \"kubernetes.io/projected/e2c4d6f2-ccde-4c36-a612-9f26e21947e1-kube-api-access-nhb5v\") pod \"e2c4d6f2-ccde-4c36-a612-9f26e21947e1\" (UID: \"e2c4d6f2-ccde-4c36-a612-9f26e21947e1\") " Oct 30 06:31:33.200230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.200204 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c4d6f2-ccde-4c36-a612-9f26e21947e1-kube-api-access-nhb5v" pod "e2c4d6f2-ccde-4c36-a612-9f26e21947e1" (UID: "e2c4d6f2-ccde-4c36-a612-9f26e21947e1"). InnerVolumeSpecName "kube-api-access-nhb5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:33.300326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.300275 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhb5v\" (UniqueName: \"kubernetes.io/projected/e2c4d6f2-ccde-4c36-a612-9f26e21947e1-kube-api-access-nhb5v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:33.792378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.792342 5001 generic.go:356] "Generic (PLEG): container finished" podID="e2c4d6f2-ccde-4c36-a612-9f26e21947e1" containerID="7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d" exitCode=137 Oct 30 06:31:33.792568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.792405 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7324-6ab4a8" Oct 30 06:31:33.792568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.792451 5001 scope.go:122] "RemoveContainer" containerID="7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d" Oct 30 06:31:33.795191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.795167 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" event={"ID":"75ca4d25-1038-4aaa-9d74-436b3222223c","Type":"ContainerStarted","Data":"2e78a0807bb4d0fbc21fdf342360b4118d5881bbe47f5789e89cbb7d93d6bebe"} Oct 30 06:31:33.795459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.795421 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" event={"ID":"75ca4d25-1038-4aaa-9d74-436b3222223c","Type":"ContainerStarted","Data":"7d4a9e26f0329c4acbda9aa2ee66f551797dcbfd469875ca49dcf0ac318e7657"} Oct 30 06:31:33.805582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.805565 5001 scope.go:122] "RemoveContainer" containerID="7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d" Oct 30 06:31:33.805867 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:31:33.805848 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d\": container with ID starting with 7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d not found: ID does not exist" containerID="7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d" Oct 30 06:31:33.805924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.805877 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d"} err="failed to get container status \"7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d\": rpc error: code = NotFound desc = could not find container \"7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d\": container with ID starting with 7eefd98252770917dcbfc0a91cbe5057a2d1580ae1cf59042b8efdfd4b16ef0d not found: ID does not exist" Oct 30 06:31:33.850324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:33.850252 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" podStartSLOduration=2.850232924 podStartE2EDuration="2.850232924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:33.849645717 +0000 UTC m=+2307.845602726" watchObservedRunningTime="2026-10-30 06:31:33.850232924 +0000 UTC m=+2307.846189934" Oct 30 06:31:33.862691 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:31:33.862652 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7324-6ab4a8\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" podUID="e2c4d6f2-ccde-4c36-a612-9f26e21947e1" pod="opendatahub/test-curl-7324-6ab4a8" Oct 30 06:31:34.147329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:34.147123 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c4d6f2-ccde-4c36-a612-9f26e21947e1" path="/var/lib/kubelet/pods/e2c4d6f2-ccde-4c36-a612-9f26e21947e1/volumes" Oct 30 06:31:34.410922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:34.410831 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:31:34.459978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:34.459944 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:34.467202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:34.467169 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:34.821539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:34.820972 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:34.827394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:34.823822 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:31:37.847926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:37.847878 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26"] Oct 30 06:31:37.848379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:37.848100 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" podUID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" containerName="istio-proxy" containerID="cri-o://a80c2cacb8f75b5fce2b69b90943b8a309448ba2aa3319ceda0dfd41e075c37b" gracePeriod=30 Oct 30 06:31:37.860391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:37.860340 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l"] Oct 30 06:31:37.860629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:37.860593 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" podUID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" containerName="istio-proxy" containerID="cri-o://9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9" gracePeriod=30 Oct 30 06:31:37.866927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:37.866896 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz"] Oct 30 06:31:37.867156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:37.867132 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" podUID="1f26970b-99c6-408d-be28-3a0938db99b6" containerName="istio-proxy" containerID="cri-o://b7012eaaf3159d9a8fb6fc5a63d08ed8891ea16c729df1794612ac295c1957ca" gracePeriod=30 Oct 30 06:31:38.197383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.197334 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v"] Oct 30 06:31:38.203627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.203604 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:31:38.218385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.218288 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-b8ca49\"" Oct 30 06:31:38.222410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.222384 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v"] Oct 30 06:31:38.281983 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:31:38.281914 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:31:38.313697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.313652 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twk9g\" (UniqueName: \"kubernetes.io/projected/b65ca078-eaa5-4bf4-a8c1-0856876434f1-kube-api-access-twk9g\") pod \"payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v\" (UID: \"b65ca078-eaa5-4bf4-a8c1-0856876434f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:31:38.313895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.313809 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65ca078-eaa5-4bf4-a8c1-0856876434f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v\" (UID: \"b65ca078-eaa5-4bf4-a8c1-0856876434f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:31:38.394329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.391994 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b"] Oct 30 06:31:38.483381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.482735 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twk9g\" (UniqueName: \"kubernetes.io/projected/b65ca078-eaa5-4bf4-a8c1-0856876434f1-kube-api-access-twk9g\") pod \"payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v\" (UID: \"b65ca078-eaa5-4bf4-a8c1-0856876434f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:31:38.483381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.482966 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65ca078-eaa5-4bf4-a8c1-0856876434f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v\" (UID: \"b65ca078-eaa5-4bf4-a8c1-0856876434f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:31:38.491332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.488635 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:31:38.599342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.599044 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-b8ca49-dockercfg-r6jth\"" Oct 30 06:31:38.693654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.672421 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twk9g\" (UniqueName: \"kubernetes.io/projected/b65ca078-eaa5-4bf4-a8c1-0856876434f1-kube-api-access-twk9g\") pod \"payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v\" (UID: \"b65ca078-eaa5-4bf4-a8c1-0856876434f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:31:38.693654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.678173 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65ca078-eaa5-4bf4-a8c1-0856876434f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v\" (UID: \"b65ca078-eaa5-4bf4-a8c1-0856876434f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:31:38.693654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.686706 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b"] Oct 30 06:31:38.712436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.699209 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f1a036-b3e2-44e7-a082-862bf2070478-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b\" (UID: \"63f1a036-b3e2-44e7-a082-862bf2070478\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:31:38.712436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.699737 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkd52\" (UniqueName: \"kubernetes.io/projected/63f1a036-b3e2-44e7-a082-862bf2070478-kube-api-access-jkd52\") pod \"payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b\" (UID: \"63f1a036-b3e2-44e7-a082-862bf2070478\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:31:38.804603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.804512 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkd52\" (UniqueName: \"kubernetes.io/projected/63f1a036-b3e2-44e7-a082-862bf2070478-kube-api-access-jkd52\") pod \"payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b\" (UID: \"63f1a036-b3e2-44e7-a082-862bf2070478\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:31:38.804769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.804681 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f1a036-b3e2-44e7-a082-862bf2070478-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b\" (UID: \"63f1a036-b3e2-44e7-a082-862bf2070478\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:31:38.805904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.805882 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f1a036-b3e2-44e7-a082-862bf2070478-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b\" (UID: \"63f1a036-b3e2-44e7-a082-862bf2070478\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:31:38.819825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.819801 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:31:38.827573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:38.827546 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkd52\" (UniqueName: \"kubernetes.io/projected/63f1a036-b3e2-44e7-a082-862bf2070478-kube-api-access-jkd52\") pod \"payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b\" (UID: \"63f1a036-b3e2-44e7-a082-862bf2070478\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:31:39.036757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.036712 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:31:39.147903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.147863 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v"] Oct 30 06:31:39.352300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.352269 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b"] Oct 30 06:31:39.354556 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:31:39.354504 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f1a036_b3e2_44e7_a082_862bf2070478.slice/crio-e912d4654de52a5d656e98855527e66d43f4a35a098f2850c39a656e202d6874 WatchSource:0}: Error finding container e912d4654de52a5d656e98855527e66d43f4a35a098f2850c39a656e202d6874: Status 404 returned error can't find the container with id e912d4654de52a5d656e98855527e66d43f4a35a098f2850c39a656e202d6874 Oct 30 06:31:39.899269 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.899230 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v"] Oct 30 06:31:39.905448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.905415 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:31:39.923508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.923480 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2d2923-dockercfg-kkljv\"" Oct 30 06:31:39.924035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.924009 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v"] Oct 30 06:31:39.930829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.930496 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" event={"ID":"63f1a036-b3e2-44e7-a082-862bf2070478","Type":"ContainerStarted","Data":"8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab"} Oct 30 06:31:39.930829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.930555 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" event={"ID":"63f1a036-b3e2-44e7-a082-862bf2070478","Type":"ContainerStarted","Data":"e912d4654de52a5d656e98855527e66d43f4a35a098f2850c39a656e202d6874"} Oct 30 06:31:39.930829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.930687 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:31:39.932772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.932745 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" event={"ID":"b65ca078-eaa5-4bf4-a8c1-0856876434f1","Type":"ContainerStarted","Data":"7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204"} Oct 30 06:31:39.932918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.932798 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:31:39.932918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.932816 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" event={"ID":"b65ca078-eaa5-4bf4-a8c1-0856876434f1","Type":"ContainerStarted","Data":"5cd6472dd826002cd3a214491565013c069a914f628ff5d2d88202a27e76f532"} Oct 30 06:31:39.992533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:39.984258 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2d2923\"" Oct 30 06:31:40.049377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:40.048547 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v\" (UID: \"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:31:40.049377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:40.048873 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r96l\" (UniqueName: \"kubernetes.io/projected/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-kube-api-access-9r96l\") pod \"payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v\" (UID: \"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:31:40.127265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:40.127200 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" podStartSLOduration=2.127179061 podStartE2EDuration="2.127179061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:39.985412387 +0000 UTC m=+2313.981369398" watchObservedRunningTime="2026-10-30 06:31:40.127179061 +0000 UTC m=+2314.123136072" Oct 30 06:31:40.169393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:40.155319 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r96l\" (UniqueName: \"kubernetes.io/projected/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-kube-api-access-9r96l\") pod \"payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v\" (UID: \"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:31:40.169393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:40.155522 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v\" (UID: \"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:31:40.169393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:40.156652 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v\" (UID: \"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:31:40.170663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:40.170584 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" podStartSLOduration=2.17056548 podStartE2EDuration="2.17056548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:40.092980947 +0000 UTC m=+2314.088937954" watchObservedRunningTime="2026-10-30 06:31:40.17056548 +0000 UTC m=+2314.166522491" Oct 30 06:31:40.278394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:40.274327 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r96l\" (UniqueName: \"kubernetes.io/projected/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-kube-api-access-9r96l\") pod \"payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v\" (UID: \"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:31:40.527361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:40.527247 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:31:40.852148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:40.852121 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v"] Oct 30 06:31:40.855754 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:31:40.855727 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb1ebd4_bede_4c40_9e3e_5f689e995b9e.slice/crio-2a5daf115dc0d04e3d1c730456ff987aaae39fe5a6f3fc60012dcd4fdba7a0e3 WatchSource:0}: Error finding container 2a5daf115dc0d04e3d1c730456ff987aaae39fe5a6f3fc60012dcd4fdba7a0e3: Status 404 returned error can't find the container with id 2a5daf115dc0d04e3d1c730456ff987aaae39fe5a6f3fc60012dcd4fdba7a0e3 Oct 30 06:31:40.945551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:40.945519 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" event={"ID":"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e","Type":"ContainerStarted","Data":"2a5daf115dc0d04e3d1c730456ff987aaae39fe5a6f3fc60012dcd4fdba7a0e3"} Oct 30 06:31:41.957840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:41.957805 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" event={"ID":"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e","Type":"ContainerStarted","Data":"577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6"} Oct 30 06:31:41.958411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:41.957867 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:31:42.005332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:42.000106 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" podStartSLOduration=3.00007957 podStartE2EDuration="3.00007957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:41.997285294 +0000 UTC m=+2315.993242302" watchObservedRunningTime="2026-10-30 06:31:42.00007957 +0000 UTC m=+2315.996036638" Oct 30 06:31:42.970695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:42.970668 5001 generic.go:356] "Generic (PLEG): container finished" podID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" containerID="a80c2cacb8f75b5fce2b69b90943b8a309448ba2aa3319ceda0dfd41e075c37b" exitCode=0 Oct 30 06:31:42.971084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:42.970743 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" event={"ID":"0d5d02f0-ca64-4cbf-be9b-7003996c44d8","Type":"ContainerDied","Data":"a80c2cacb8f75b5fce2b69b90943b8a309448ba2aa3319ceda0dfd41e075c37b"} Oct 30 06:31:42.973233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:42.973209 5001 generic.go:356] "Generic (PLEG): container finished" podID="1f26970b-99c6-408d-be28-3a0938db99b6" containerID="b7012eaaf3159d9a8fb6fc5a63d08ed8891ea16c729df1794612ac295c1957ca" exitCode=0 Oct 30 06:31:42.973393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:42.973299 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" event={"ID":"1f26970b-99c6-408d-be28-3a0938db99b6","Type":"ContainerDied","Data":"b7012eaaf3159d9a8fb6fc5a63d08ed8891ea16c729df1794612ac295c1957ca"} Oct 30 06:31:43.181665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.181634 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:31:43.218522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.218494 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:31:43.236995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.236969 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:31:43.257801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.257766 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-data\") pod \"1f26970b-99c6-408d-be28-3a0938db99b6\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " Oct 30 06:31:43.258396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.258371 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-socket\") pod \"1f26970b-99c6-408d-be28-3a0938db99b6\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " Oct 30 06:31:43.260390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.258893 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-certs\") pod \"1f26970b-99c6-408d-be28-3a0938db99b6\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " Oct 30 06:31:43.260390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.258931 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f26970b-99c6-408d-be28-3a0938db99b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f26970b-99c6-408d-be28-3a0938db99b6-istio-podinfo\") pod \"1f26970b-99c6-408d-be28-3a0938db99b6\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " Oct 30 06:31:43.260390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.258979 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f26970b-99c6-408d-be28-3a0938db99b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f26970b-99c6-408d-be28-3a0938db99b6-istiod-ca-cert\") pod \"1f26970b-99c6-408d-be28-3a0938db99b6\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " Oct 30 06:31:43.260390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.259081 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-kube-api-access-zrcmx\" (UniqueName: \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-kube-api-access-zrcmx\") pod \"1f26970b-99c6-408d-be28-3a0938db99b6\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " Oct 30 06:31:43.260390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.259254 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-envoy\") pod \"1f26970b-99c6-408d-be28-3a0938db99b6\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " Oct 30 06:31:43.260390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.259382 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-credential-socket\") pod \"1f26970b-99c6-408d-be28-3a0938db99b6\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " Oct 30 06:31:43.260390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.259414 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-istio-token\" (UniqueName: \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-istio-token\") pod \"1f26970b-99c6-408d-be28-3a0938db99b6\" (UID: \"1f26970b-99c6-408d-be28-3a0938db99b6\") " Oct 30 06:31:43.262614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.258259 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-data" pod "1f26970b-99c6-408d-be28-3a0938db99b6" (UID: "1f26970b-99c6-408d-be28-3a0938db99b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.262614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.258809 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-socket" pod "1f26970b-99c6-408d-be28-3a0938db99b6" (UID: "1f26970b-99c6-408d-be28-3a0938db99b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.262614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.262354 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f26970b-99c6-408d-be28-3a0938db99b6-istiod-ca-cert" pod "1f26970b-99c6-408d-be28-3a0938db99b6" (UID: "1f26970b-99c6-408d-be28-3a0938db99b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:43.263213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.262893 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-certs" pod "1f26970b-99c6-408d-be28-3a0938db99b6" (UID: "1f26970b-99c6-408d-be28-3a0938db99b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.263350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.263292 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-credential-socket" pod "1f26970b-99c6-408d-be28-3a0938db99b6" (UID: "1f26970b-99c6-408d-be28-3a0938db99b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.271831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.271700 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-envoy" pod "1f26970b-99c6-408d-be28-3a0938db99b6" (UID: "1f26970b-99c6-408d-be28-3a0938db99b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.271831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.271738 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-kube-api-access-zrcmx" pod "1f26970b-99c6-408d-be28-3a0938db99b6" (UID: "1f26970b-99c6-408d-be28-3a0938db99b6"). InnerVolumeSpecName "kube-api-access-zrcmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:43.271831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.271795 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f26970b-99c6-408d-be28-3a0938db99b6-istio-podinfo" pod "1f26970b-99c6-408d-be28-3a0938db99b6" (UID: "1f26970b-99c6-408d-be28-3a0938db99b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:31:43.273780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.273749 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-istio-token" pod "1f26970b-99c6-408d-be28-3a0938db99b6" (UID: "1f26970b-99c6-408d-be28-3a0938db99b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:43.361075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361022 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-credential-socket\") pod \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " Oct 30 06:31:43.361075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361076 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-socket\") pod \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " Oct 30 06:31:43.361362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361139 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-kube-api-access-vbhx6\" (UniqueName: \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-kube-api-access-vbhx6\") pod \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " Oct 30 06:31:43.361362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361182 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-podinfo\") pod \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " Oct 30 06:31:43.361362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361235 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istiod-ca-cert\") pod \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " Oct 30 06:31:43.361362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361267 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-podinfo\") pod \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " Oct 30 06:31:43.361595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361382 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-data\") pod \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " Oct 30 06:31:43.361595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361431 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-kube-api-access-pr7kn\" (UniqueName: \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-kube-api-access-pr7kn\") pod \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " Oct 30 06:31:43.361595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361475 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-data\") pod \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " Oct 30 06:31:43.361595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361542 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istiod-ca-cert\") pod \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " Oct 30 06:31:43.361810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361619 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-certs\") pod \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " Oct 30 06:31:43.361810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361683 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-socket\") pod \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " Oct 30 06:31:43.361810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361721 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-credential-socket\") pod \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " Oct 30 06:31:43.361810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361783 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-certs\") pod \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " Oct 30 06:31:43.362011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361826 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-token\" (UniqueName: \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-token\") pod \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " Oct 30 06:31:43.362011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361947 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-token\" (UniqueName: \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-token\") pod \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " Oct 30 06:31:43.362011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.361995 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-envoy\") pod \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\" (UID: \"0d5d02f0-ca64-4cbf-be9b-7003996c44d8\") " Oct 30 06:31:43.362154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362043 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-envoy\") pod \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\" (UID: \"5c3a6b7c-98e8-4192-aad0-1b917966ffed\") " Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362511 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-credential-socket" pod "0d5d02f0-ca64-4cbf-be9b-7003996c44d8" (UID: "0d5d02f0-ca64-4cbf-be9b-7003996c44d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362770 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-certs" pod "5c3a6b7c-98e8-4192-aad0-1b917966ffed" (UID: "5c3a6b7c-98e8-4192-aad0-1b917966ffed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362839 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362862 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362875 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362888 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362900 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362912 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f26970b-99c6-408d-be28-3a0938db99b6-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362924 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f26970b-99c6-408d-be28-3a0938db99b6-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362937 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrcmx\" (UniqueName: \"kubernetes.io/projected/1f26970b-99c6-408d-be28-3a0938db99b6-kube-api-access-zrcmx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362950 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f26970b-99c6-408d-be28-3a0938db99b6-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.362962 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.363184 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-socket" pod "5c3a6b7c-98e8-4192-aad0-1b917966ffed" (UID: "5c3a6b7c-98e8-4192-aad0-1b917966ffed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.363483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.363444 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-socket" pod "0d5d02f0-ca64-4cbf-be9b-7003996c44d8" (UID: "0d5d02f0-ca64-4cbf-be9b-7003996c44d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.364766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.363912 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istiod-ca-cert" pod "0d5d02f0-ca64-4cbf-be9b-7003996c44d8" (UID: "0d5d02f0-ca64-4cbf-be9b-7003996c44d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:43.364766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.364440 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-credential-socket" pod "5c3a6b7c-98e8-4192-aad0-1b917966ffed" (UID: "5c3a6b7c-98e8-4192-aad0-1b917966ffed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.365247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.365101 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-data" pod "5c3a6b7c-98e8-4192-aad0-1b917966ffed" (UID: "5c3a6b7c-98e8-4192-aad0-1b917966ffed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.365613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.365366 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-envoy" pod "5c3a6b7c-98e8-4192-aad0-1b917966ffed" (UID: "5c3a6b7c-98e8-4192-aad0-1b917966ffed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.365613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.365407 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-data" pod "0d5d02f0-ca64-4cbf-be9b-7003996c44d8" (UID: "0d5d02f0-ca64-4cbf-be9b-7003996c44d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.366088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.365892 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istiod-ca-cert" pod "5c3a6b7c-98e8-4192-aad0-1b917966ffed" (UID: "5c3a6b7c-98e8-4192-aad0-1b917966ffed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:43.366240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.366209 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-certs" pod "0d5d02f0-ca64-4cbf-be9b-7003996c44d8" (UID: "0d5d02f0-ca64-4cbf-be9b-7003996c44d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.369200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.369174 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-podinfo" pod "5c3a6b7c-98e8-4192-aad0-1b917966ffed" (UID: "5c3a6b7c-98e8-4192-aad0-1b917966ffed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:31:43.369763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.369737 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-kube-api-access-pr7kn" pod "5c3a6b7c-98e8-4192-aad0-1b917966ffed" (UID: "5c3a6b7c-98e8-4192-aad0-1b917966ffed"). InnerVolumeSpecName "kube-api-access-pr7kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:43.372851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.372774 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-podinfo" pod "0d5d02f0-ca64-4cbf-be9b-7003996c44d8" (UID: "0d5d02f0-ca64-4cbf-be9b-7003996c44d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:31:43.372851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.372800 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-token" pod "0d5d02f0-ca64-4cbf-be9b-7003996c44d8" (UID: "0d5d02f0-ca64-4cbf-be9b-7003996c44d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:43.372851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.372830 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-kube-api-access-vbhx6" pod "0d5d02f0-ca64-4cbf-be9b-7003996c44d8" (UID: "0d5d02f0-ca64-4cbf-be9b-7003996c44d8"). InnerVolumeSpecName "kube-api-access-vbhx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:43.373057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.372901 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-token" pod "5c3a6b7c-98e8-4192-aad0-1b917966ffed" (UID: "5c3a6b7c-98e8-4192-aad0-1b917966ffed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:43.374513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.374491 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-envoy" pod "0d5d02f0-ca64-4cbf-be9b-7003996c44d8" (UID: "0d5d02f0-ca64-4cbf-be9b-7003996c44d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:43.464386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464268 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464340 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464350 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464358 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464368 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr7kn\" (UniqueName: \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-kube-api-access-pr7kn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464379 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464388 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464396 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464404 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464412 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464421 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464428 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464435 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464442 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464451 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3a6b7c-98e8-4192-aad0-1b917966ffed-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464459 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.464977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.464469 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbhx6\" (UniqueName: \"kubernetes.io/projected/0d5d02f0-ca64-4cbf-be9b-7003996c44d8-kube-api-access-vbhx6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:31:43.985454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.985416 5001 generic.go:356] "Generic (PLEG): container finished" podID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" containerID="9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9" exitCode=0 Oct 30 06:31:43.985929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.985465 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" Oct 30 06:31:43.985929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.985507 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" event={"ID":"5c3a6b7c-98e8-4192-aad0-1b917966ffed","Type":"ContainerDied","Data":"9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9"} Oct 30 06:31:43.985929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.985574 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l" event={"ID":"5c3a6b7c-98e8-4192-aad0-1b917966ffed","Type":"ContainerDied","Data":"999b5f80c909ffaf706a6ffb5f54bdfd526569582ecba3a277968e0f9b734de5"} Oct 30 06:31:43.985929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.985607 5001 scope.go:122] "RemoveContainer" containerID="9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9" Oct 30 06:31:43.987744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.987723 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" Oct 30 06:31:43.988236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.988154 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26" event={"ID":"0d5d02f0-ca64-4cbf-be9b-7003996c44d8","Type":"ContainerDied","Data":"9cb08173803ff048bb261d748c70875ffcd091c48758cbf518e39b5fc73148a0"} Oct 30 06:31:43.989815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.989792 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" event={"ID":"1f26970b-99c6-408d-be28-3a0938db99b6","Type":"ContainerDied","Data":"a1b4ba6cf0b74898614a1cd39ae9600828c550f2281b6772af637b9409d2b958"} Oct 30 06:31:43.989902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:43.989882 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz" Oct 30 06:31:44.000686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.000647 5001 scope.go:122] "RemoveContainer" containerID="9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9" Oct 30 06:31:44.001073 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:31:44.001049 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9\": container with ID starting with 9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9 not found: ID does not exist" containerID="9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9" Oct 30 06:31:44.001143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.001083 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9"} err="failed to get container status \"9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9\": rpc error: code = NotFound desc = could not find container \"9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9\": container with ID starting with 9cc86b6949f6d4ffa8503f0bd3642e8a023b7c0c84233f82c4e3933c43b19ff9 not found: ID does not exist" Oct 30 06:31:44.001143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.001099 5001 scope.go:122] "RemoveContainer" containerID="a80c2cacb8f75b5fce2b69b90943b8a309448ba2aa3319ceda0dfd41e075c37b" Oct 30 06:31:44.012830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.012812 5001 scope.go:122] "RemoveContainer" containerID="b7012eaaf3159d9a8fb6fc5a63d08ed8891ea16c729df1794612ac295c1957ca" Oct 30 06:31:44.038261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.038226 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l"] Oct 30 06:31:44.050430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.050400 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-cwk4l"] Oct 30 06:31:44.105296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.105263 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz"] Oct 30 06:31:44.109510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.109481 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-gtbzz"] Oct 30 06:31:44.119343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.119320 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f26970b-99c6-408d-be28-3a0938db99b6" path="/var/lib/kubelet/pods/1f26970b-99c6-408d-be28-3a0938db99b6/volumes" Oct 30 06:31:44.119915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.119892 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" path="/var/lib/kubelet/pods/5c3a6b7c-98e8-4192-aad0-1b917966ffed/volumes" Oct 30 06:31:44.140286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.140212 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26"] Oct 30 06:31:44.144448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:44.144424 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-65q26"] Oct 30 06:31:46.119836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:46.119799 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" path="/var/lib/kubelet/pods/0d5d02f0-ca64-4cbf-be9b-7003996c44d8/volumes" Oct 30 06:31:50.949114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:50.949075 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:31:50.949750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:50.949669 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:31:52.977551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:31:52.977521 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:32:05.672004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.671969 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg"] Oct 30 06:32:05.672701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.672685 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1f26970b-99c6-408d-be28-3a0938db99b6" containerName="istio-proxy" Oct 30 06:32:05.672763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.672703 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f26970b-99c6-408d-be28-3a0938db99b6" containerName="istio-proxy" Oct 30 06:32:05.672763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.672737 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" containerName="istio-proxy" Oct 30 06:32:05.672763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.672742 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" containerName="istio-proxy" Oct 30 06:32:05.672763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.672754 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" containerName="istio-proxy" Oct 30 06:32:05.672763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.672759 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" containerName="istio-proxy" Oct 30 06:32:05.672936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.672861 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3a6b7c-98e8-4192-aad0-1b917966ffed" containerName="istio-proxy" Oct 30 06:32:05.672936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.672871 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5d02f0-ca64-4cbf-be9b-7003996c44d8" containerName="istio-proxy" Oct 30 06:32:05.672936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.672881 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f26970b-99c6-408d-be28-3a0938db99b6" containerName="istio-proxy" Oct 30 06:32:05.676656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.676633 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:05.694341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.694297 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg"] Oct 30 06:32:05.701200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.701173 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:32:05.701927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.701904 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:32:05.702038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.702006 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:32:05.702038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.702035 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:32:05.763981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.763940 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10a218b-9slwg\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:05.764185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.764097 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10a218b-9slwg\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:05.764295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.764271 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcxhp\" (UniqueName: \"kubernetes.io/projected/592b6f9a-cfd5-4f7f-a50a-3da42e914750-kube-api-access-wcxhp\") pod \"maas-api-revoke-sub-c10a218b-9slwg\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:05.866263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.866222 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10a218b-9slwg\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:05.866480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.866389 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcxhp\" (UniqueName: \"kubernetes.io/projected/592b6f9a-cfd5-4f7f-a50a-3da42e914750-kube-api-access-wcxhp\") pod \"maas-api-revoke-sub-c10a218b-9slwg\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:05.866537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.866510 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10a218b-9slwg\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:05.867096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.867058 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10a218b-9slwg\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:05.869149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.869125 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10a218b-9slwg\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:05.889428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.889398 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcxhp\" (UniqueName: \"kubernetes.io/projected/592b6f9a-cfd5-4f7f-a50a-3da42e914750-kube-api-access-wcxhp\") pod \"maas-api-revoke-sub-c10a218b-9slwg\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:05.984722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.984630 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw"] Oct 30 06:32:05.989899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.989877 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:05.992433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:05.992410 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:06.009548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.009512 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw"] Oct 30 06:32:06.070644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.069935 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be67b9d1-s6flw\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:06.070644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.069983 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmlhd\" (UniqueName: \"kubernetes.io/projected/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-kube-api-access-kmlhd\") pod \"maas-api-revoke-sub-be67b9d1-s6flw\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:06.070644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.070054 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be67b9d1-s6flw\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:06.171776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.171740 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be67b9d1-s6flw\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:06.172032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.171790 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmlhd\" (UniqueName: \"kubernetes.io/projected/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-kube-api-access-kmlhd\") pod \"maas-api-revoke-sub-be67b9d1-s6flw\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:06.172856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.172123 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be67b9d1-s6flw\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:06.173606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.173580 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be67b9d1-s6flw\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:06.174795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.174770 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be67b9d1-s6flw\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:06.195139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.195111 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmlhd\" (UniqueName: \"kubernetes.io/projected/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-kube-api-access-kmlhd\") pod \"maas-api-revoke-sub-be67b9d1-s6flw\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:06.308829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.308744 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:06.323189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.323151 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg"] Oct 30 06:32:06.328032 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:32:06.328003 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592b6f9a_cfd5_4f7f_a50a_3da42e914750.slice/crio-34f042cf390fa0bc4fc23408e1553a78a743672ffb1b8e7a7b1573969df87585 WatchSource:0}: Error finding container 34f042cf390fa0bc4fc23408e1553a78a743672ffb1b8e7a7b1573969df87585: Status 404 returned error can't find the container with id 34f042cf390fa0bc4fc23408e1553a78a743672ffb1b8e7a7b1573969df87585 Oct 30 06:32:06.629522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:06.629489 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw"] Oct 30 06:32:06.630567 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:32:06.630538 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ead8c6_41c8_40e6_9fe7_6dedaaeab73f.slice/crio-19075d74bc3533a02e58b46762d9833f2e7897e5068fe95a36a7c90f47731e1d WatchSource:0}: Error finding container 19075d74bc3533a02e58b46762d9833f2e7897e5068fe95a36a7c90f47731e1d: Status 404 returned error can't find the container with id 19075d74bc3533a02e58b46762d9833f2e7897e5068fe95a36a7c90f47731e1d Oct 30 06:32:07.213136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:07.212490 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" event={"ID":"592b6f9a-cfd5-4f7f-a50a-3da42e914750","Type":"ContainerStarted","Data":"729170e252b2d1cf77f627e30df79a6e35dce4a07573e1c0ae8ceb657f5d78e5"} Oct 30 06:32:07.213136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:07.212559 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" event={"ID":"592b6f9a-cfd5-4f7f-a50a-3da42e914750","Type":"ContainerStarted","Data":"34f042cf390fa0bc4fc23408e1553a78a743672ffb1b8e7a7b1573969df87585"} Oct 30 06:32:07.214354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:07.214327 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" event={"ID":"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f","Type":"ContainerStarted","Data":"b34d56118d91d82c526fd4b50d9dabbcaee9cc1c3ac7002cc2e013eef44faa9f"} Oct 30 06:32:07.214472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:07.214374 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" event={"ID":"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f","Type":"ContainerStarted","Data":"19075d74bc3533a02e58b46762d9833f2e7897e5068fe95a36a7c90f47731e1d"} Oct 30 06:32:07.254958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:07.254893 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" podStartSLOduration=2.254875506 podStartE2EDuration="2.254875506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:07.251063799 +0000 UTC m=+2341.247020823" watchObservedRunningTime="2026-10-30 06:32:07.254875506 +0000 UTC m=+2341.250832520" Oct 30 06:32:07.292206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:07.292162 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" podStartSLOduration=2.2921478 podStartE2EDuration="2.2921478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:07.289433419 +0000 UTC m=+2341.285390431" watchObservedRunningTime="2026-10-30 06:32:07.2921478 +0000 UTC m=+2341.288104809" Oct 30 06:32:08.225044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:08.225007 5001 generic.go:356] "Generic (PLEG): container finished" podID="592b6f9a-cfd5-4f7f-a50a-3da42e914750" containerID="729170e252b2d1cf77f627e30df79a6e35dce4a07573e1c0ae8ceb657f5d78e5" exitCode=0 Oct 30 06:32:08.225579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:08.225058 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" event={"ID":"592b6f9a-cfd5-4f7f-a50a-3da42e914750","Type":"ContainerDied","Data":"729170e252b2d1cf77f627e30df79a6e35dce4a07573e1c0ae8ceb657f5d78e5"} Oct 30 06:32:08.227022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:08.227002 5001 generic.go:356] "Generic (PLEG): container finished" podID="43ead8c6-41c8-40e6-9fe7-6dedaaeab73f" containerID="b34d56118d91d82c526fd4b50d9dabbcaee9cc1c3ac7002cc2e013eef44faa9f" exitCode=0 Oct 30 06:32:08.227166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:08.227090 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" event={"ID":"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f","Type":"ContainerDied","Data":"b34d56118d91d82c526fd4b50d9dabbcaee9cc1c3ac7002cc2e013eef44faa9f"} Oct 30 06:32:08.267283 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:32:08.267248 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:32:09.423184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.423156 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:09.494811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.494700 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:09.533354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.533321 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-openshift-ca\") pod \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " Oct 30 06:32:09.533354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.533359 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-openshift-ca\") pod \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " Oct 30 06:32:09.533638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.533434 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-xks-ca\") pod \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " Oct 30 06:32:09.533638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.533480 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592b6f9a-cfd5-4f7f-a50a-3da42e914750-kube-api-access-wcxhp\" (UniqueName: \"kubernetes.io/projected/592b6f9a-cfd5-4f7f-a50a-3da42e914750-kube-api-access-wcxhp\") pod \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " Oct 30 06:32:09.533638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.533499 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-kube-api-access-kmlhd\" (UniqueName: \"kubernetes.io/projected/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-kube-api-access-kmlhd\") pod \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\" (UID: \"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f\") " Oct 30 06:32:09.533638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.533555 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-xks-ca\") pod \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\" (UID: \"592b6f9a-cfd5-4f7f-a50a-3da42e914750\") " Oct 30 06:32:09.533920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.533890 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-openshift-ca" pod "592b6f9a-cfd5-4f7f-a50a-3da42e914750" (UID: "592b6f9a-cfd5-4f7f-a50a-3da42e914750"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:32:09.533989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.533893 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-openshift-ca" pod "43ead8c6-41c8-40e6-9fe7-6dedaaeab73f" (UID: "43ead8c6-41c8-40e6-9fe7-6dedaaeab73f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:32:09.534400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.534373 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:09.534538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.534404 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:09.536720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.536693 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-xks-ca" pod "43ead8c6-41c8-40e6-9fe7-6dedaaeab73f" (UID: "43ead8c6-41c8-40e6-9fe7-6dedaaeab73f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:32:09.538194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.538166 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592b6f9a-cfd5-4f7f-a50a-3da42e914750-kube-api-access-wcxhp" pod "592b6f9a-cfd5-4f7f-a50a-3da42e914750" (UID: "592b6f9a-cfd5-4f7f-a50a-3da42e914750"). InnerVolumeSpecName "kube-api-access-wcxhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:09.542954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.542928 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-kube-api-access-kmlhd" pod "43ead8c6-41c8-40e6-9fe7-6dedaaeab73f" (UID: "43ead8c6-41c8-40e6-9fe7-6dedaaeab73f"). InnerVolumeSpecName "kube-api-access-kmlhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:09.545817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.545790 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-xks-ca" pod "592b6f9a-cfd5-4f7f-a50a-3da42e914750" (UID: "592b6f9a-cfd5-4f7f-a50a-3da42e914750"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:32:09.636029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.635997 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:09.636029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.636026 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcxhp\" (UniqueName: \"kubernetes.io/projected/592b6f9a-cfd5-4f7f-a50a-3da42e914750-kube-api-access-wcxhp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:09.636029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.636037 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmlhd\" (UniqueName: \"kubernetes.io/projected/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f-kube-api-access-kmlhd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:09.636285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.636047 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592b6f9a-cfd5-4f7f-a50a-3da42e914750-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:09.707127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.707092 5001 scope.go:122] "RemoveContainer" containerID="432bf344ef1df24dacfa6600977ce5d18aded6475cb49baf850be816f291fffb" Oct 30 06:32:09.718957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.718921 5001 scope.go:122] "RemoveContainer" containerID="690a07e627efef9b1a6b24fb0fde02898d9667fa0ebd1a29a0bbcb06394a3987" Oct 30 06:32:09.730279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.730256 5001 scope.go:122] "RemoveContainer" containerID="ef44f7c5f11f7eb5b9fb637924b96e72db0211fdb9aea5504f7576d2b046c9ac" Oct 30 06:32:09.741469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.741446 5001 scope.go:122] "RemoveContainer" containerID="bc6fa23539b045905e60ae2352fa8dd2116265168fa72c1590042aeb1e817940" Oct 30 06:32:09.752432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.752410 5001 scope.go:122] "RemoveContainer" containerID="9ab955072530ec810cae1b271d81b59e37b55aa6fe6ea5976af6ffd7dc24ec49" Oct 30 06:32:09.763369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.763318 5001 scope.go:122] "RemoveContainer" containerID="e2005bf97877b18ea7160051545e04d0dda057c8e14c995899d1d9d27c343838" Oct 30 06:32:09.773482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.773446 5001 scope.go:122] "RemoveContainer" containerID="7d4c89a6d8e60b8494ea5a3e1d84f85df46c2e28ab0bfdb6a0c26300b8f9539d" Oct 30 06:32:09.783458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.783435 5001 scope.go:122] "RemoveContainer" containerID="731a899c21b1d73174edc68af0b62982b3812e66d35e55ea621dcbb3ddb5be95" Oct 30 06:32:09.793399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.793290 5001 scope.go:122] "RemoveContainer" containerID="830010d8f74056dcf3b66e773433e1478f1494b8e50d198edeaec19a961ace4e" Oct 30 06:32:09.803611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.803589 5001 scope.go:122] "RemoveContainer" containerID="743db714948c8407ab0b1dd12246f84d84e2227ae95571d5aca7ca0a9283ed8d" Oct 30 06:32:09.813936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:09.813910 5001 scope.go:122] "RemoveContainer" containerID="d2f7174c2222fb60822c99aa9473c07c07d0313fdc6c983adab90c0d8398411f" Oct 30 06:32:10.244939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:10.244908 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" Oct 30 06:32:10.245132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:10.244934 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg" event={"ID":"592b6f9a-cfd5-4f7f-a50a-3da42e914750","Type":"ContainerDied","Data":"34f042cf390fa0bc4fc23408e1553a78a743672ffb1b8e7a7b1573969df87585"} Oct 30 06:32:10.245132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:10.244985 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34f042cf390fa0bc4fc23408e1553a78a743672ffb1b8e7a7b1573969df87585" Oct 30 06:32:10.246411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:10.246385 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" Oct 30 06:32:10.246570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:10.246425 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw" event={"ID":"43ead8c6-41c8-40e6-9fe7-6dedaaeab73f","Type":"ContainerDied","Data":"19075d74bc3533a02e58b46762d9833f2e7897e5068fe95a36a7c90f47731e1d"} Oct 30 06:32:10.246570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:10.246465 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19075d74bc3533a02e58b46762d9833f2e7897e5068fe95a36a7c90f47731e1d" Oct 30 06:32:16.060740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.060698 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz"] Oct 30 06:32:16.061649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.061625 5001 cpu_manager.go:414] "containerMap: removing container" podUID="43ead8c6-41c8-40e6-9fe7-6dedaaeab73f" containerName="revoke-keys" Oct 30 06:32:16.061649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.061648 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ead8c6-41c8-40e6-9fe7-6dedaaeab73f" containerName="revoke-keys" Oct 30 06:32:16.061844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.061695 5001 cpu_manager.go:414] "containerMap: removing container" podUID="592b6f9a-cfd5-4f7f-a50a-3da42e914750" containerName="revoke-keys" Oct 30 06:32:16.061844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.061702 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592b6f9a-cfd5-4f7f-a50a-3da42e914750" containerName="revoke-keys" Oct 30 06:32:16.061951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.061879 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="592b6f9a-cfd5-4f7f-a50a-3da42e914750" containerName="revoke-keys" Oct 30 06:32:16.061951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.061902 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ead8c6-41c8-40e6-9fe7-6dedaaeab73f" containerName="revoke-keys" Oct 30 06:32:16.068899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.068875 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:16.086995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.086964 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz"] Oct 30 06:32:16.092752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.092604 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:32:16.092752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.092604 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:32:16.092752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.092614 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:32:16.100498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.100466 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:32:16.152629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.152596 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83ba65d-l8rlz\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:16.152826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.152807 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83ba65d-l8rlz\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:16.152993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.152977 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jgm\" (UniqueName: \"kubernetes.io/projected/19764304-b00c-446e-9b9c-b3d30e8beebf-kube-api-access-77jgm\") pod \"maas-api-revoke-sub-a83ba65d-l8rlz\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:16.255298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.255255 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jgm\" (UniqueName: \"kubernetes.io/projected/19764304-b00c-446e-9b9c-b3d30e8beebf-kube-api-access-77jgm\") pod \"maas-api-revoke-sub-a83ba65d-l8rlz\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:16.256808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.256777 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83ba65d-l8rlz\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:16.275688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.275643 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83ba65d-l8rlz\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:16.276561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.257928 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83ba65d-l8rlz\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:16.288609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.288563 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jgm\" (UniqueName: \"kubernetes.io/projected/19764304-b00c-446e-9b9c-b3d30e8beebf-kube-api-access-77jgm\") pod \"maas-api-revoke-sub-a83ba65d-l8rlz\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:16.289078 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.289048 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83ba65d-l8rlz\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:16.389191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.389150 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:16.439335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.439284 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2"] Oct 30 06:32:16.445464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.445441 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:16.473138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.473102 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2"] Oct 30 06:32:16.581003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.580946 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11a10a3-7ftt2\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:16.581563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.581484 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11a10a3-7ftt2\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:16.581882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.581822 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qthn\" (UniqueName: \"kubernetes.io/projected/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-kube-api-access-2qthn\") pod \"maas-api-revoke-sub-e11a10a3-7ftt2\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:16.683171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.683093 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11a10a3-7ftt2\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:16.683497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.683208 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qthn\" (UniqueName: \"kubernetes.io/projected/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-kube-api-access-2qthn\") pod \"maas-api-revoke-sub-e11a10a3-7ftt2\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:16.683497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.683265 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11a10a3-7ftt2\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:16.683961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.683942 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11a10a3-7ftt2\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:16.685739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.685715 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11a10a3-7ftt2\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:16.706151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.706121 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qthn\" (UniqueName: \"kubernetes.io/projected/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-kube-api-access-2qthn\") pod \"maas-api-revoke-sub-e11a10a3-7ftt2\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:16.715812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.715788 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz"] Oct 30 06:32:16.717874 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:32:16.717845 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19764304_b00c_446e_9b9c_b3d30e8beebf.slice/crio-20505f0cb46feaf9546e851e545cc26885bb73078ee5bf6127ed84c9b198ae97 WatchSource:0}: Error finding container 20505f0cb46feaf9546e851e545cc26885bb73078ee5bf6127ed84c9b198ae97: Status 404 returned error can't find the container with id 20505f0cb46feaf9546e851e545cc26885bb73078ee5bf6127ed84c9b198ae97 Oct 30 06:32:16.770721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:16.770679 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:17.085250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:17.085216 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2"] Oct 30 06:32:17.090339 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:32:17.090292 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edc91af_0a89_46a6_bd4c_cbbb4aee2336.slice/crio-1ad9ea0174e542b6c23b62d18b2e0550ef57069b9ff16a946f140a7990fbbec6 WatchSource:0}: Error finding container 1ad9ea0174e542b6c23b62d18b2e0550ef57069b9ff16a946f140a7990fbbec6: Status 404 returned error can't find the container with id 1ad9ea0174e542b6c23b62d18b2e0550ef57069b9ff16a946f140a7990fbbec6 Oct 30 06:32:17.320917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:17.320828 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" event={"ID":"19764304-b00c-446e-9b9c-b3d30e8beebf","Type":"ContainerStarted","Data":"a0ef9d5c4283b1743eabc86e6f2e10e495319312aebdeae4f2c12b49f1491521"} Oct 30 06:32:17.320917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:17.320878 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" event={"ID":"19764304-b00c-446e-9b9c-b3d30e8beebf","Type":"ContainerStarted","Data":"20505f0cb46feaf9546e851e545cc26885bb73078ee5bf6127ed84c9b198ae97"} Oct 30 06:32:17.322241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:17.322217 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" event={"ID":"6edc91af-0a89-46a6-bd4c-cbbb4aee2336","Type":"ContainerStarted","Data":"1ad9ea0174e542b6c23b62d18b2e0550ef57069b9ff16a946f140a7990fbbec6"} Oct 30 06:32:17.360670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:17.360618 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" podStartSLOduration=1.360603992 podStartE2EDuration="1.360603992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:17.359419838 +0000 UTC m=+2351.355376849" watchObservedRunningTime="2026-10-30 06:32:17.360603992 +0000 UTC m=+2351.356561002" Oct 30 06:32:18.342594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:18.342562 5001 generic.go:356] "Generic (PLEG): container finished" podID="19764304-b00c-446e-9b9c-b3d30e8beebf" containerID="a0ef9d5c4283b1743eabc86e6f2e10e495319312aebdeae4f2c12b49f1491521" exitCode=0 Oct 30 06:32:18.343031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:18.342649 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" event={"ID":"19764304-b00c-446e-9b9c-b3d30e8beebf","Type":"ContainerDied","Data":"a0ef9d5c4283b1743eabc86e6f2e10e495319312aebdeae4f2c12b49f1491521"} Oct 30 06:32:18.344583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:18.344561 5001 generic.go:356] "Generic (PLEG): container finished" podID="6edc91af-0a89-46a6-bd4c-cbbb4aee2336" containerID="b6d93db729c4a873fd663373387a63b3d6ae97d0e4e4f795684029ce5c998289" exitCode=0 Oct 30 06:32:18.344695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:18.344648 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" event={"ID":"6edc91af-0a89-46a6-bd4c-cbbb4aee2336","Type":"ContainerDied","Data":"b6d93db729c4a873fd663373387a63b3d6ae97d0e4e4f795684029ce5c998289"} Oct 30 06:32:19.552038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.551983 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:19.599671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.599643 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:19.622328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.620728 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-xks-ca\") pod \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " Oct 30 06:32:19.622328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.621452 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-openshift-ca\") pod \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " Oct 30 06:32:19.622328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.621925 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-openshift-ca" pod "6edc91af-0a89-46a6-bd4c-cbbb4aee2336" (UID: "6edc91af-0a89-46a6-bd4c-cbbb4aee2336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:32:19.622328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.621984 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19764304-b00c-446e-9b9c-b3d30e8beebf-kube-api-access-77jgm\" (UniqueName: \"kubernetes.io/projected/19764304-b00c-446e-9b9c-b3d30e8beebf-kube-api-access-77jgm\") pod \"19764304-b00c-446e-9b9c-b3d30e8beebf\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " Oct 30 06:32:19.622328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.622034 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-openshift-ca\") pod \"19764304-b00c-446e-9b9c-b3d30e8beebf\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " Oct 30 06:32:19.622328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.622062 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-kube-api-access-2qthn\" (UniqueName: \"kubernetes.io/projected/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-kube-api-access-2qthn\") pod \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\" (UID: \"6edc91af-0a89-46a6-bd4c-cbbb4aee2336\") " Oct 30 06:32:19.622328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.622091 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-xks-ca\") pod \"19764304-b00c-446e-9b9c-b3d30e8beebf\" (UID: \"19764304-b00c-446e-9b9c-b3d30e8beebf\") " Oct 30 06:32:19.622884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.622683 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-openshift-ca" pod "19764304-b00c-446e-9b9c-b3d30e8beebf" (UID: "19764304-b00c-446e-9b9c-b3d30e8beebf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:32:19.622884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.622787 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:19.624706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.624668 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-xks-ca" pod "6edc91af-0a89-46a6-bd4c-cbbb4aee2336" (UID: "6edc91af-0a89-46a6-bd4c-cbbb4aee2336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:32:19.625124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.625095 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19764304-b00c-446e-9b9c-b3d30e8beebf-kube-api-access-77jgm" pod "19764304-b00c-446e-9b9c-b3d30e8beebf" (UID: "19764304-b00c-446e-9b9c-b3d30e8beebf"). InnerVolumeSpecName "kube-api-access-77jgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:19.626199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.626176 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-xks-ca" pod "19764304-b00c-446e-9b9c-b3d30e8beebf" (UID: "19764304-b00c-446e-9b9c-b3d30e8beebf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:32:19.626354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.626290 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-kube-api-access-2qthn" pod "6edc91af-0a89-46a6-bd4c-cbbb4aee2336" (UID: "6edc91af-0a89-46a6-bd4c-cbbb4aee2336"). InnerVolumeSpecName "kube-api-access-2qthn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:19.723825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.723788 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:19.723825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.723821 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77jgm\" (UniqueName: \"kubernetes.io/projected/19764304-b00c-446e-9b9c-b3d30e8beebf-kube-api-access-77jgm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:19.723825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.723831 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:19.724098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.723842 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qthn\" (UniqueName: \"kubernetes.io/projected/6edc91af-0a89-46a6-bd4c-cbbb4aee2336-kube-api-access-2qthn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:19.724098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:19.723851 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19764304-b00c-446e-9b9c-b3d30e8beebf-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:20.362266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:20.362236 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" Oct 30 06:32:20.362494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:20.362262 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz" event={"ID":"19764304-b00c-446e-9b9c-b3d30e8beebf","Type":"ContainerDied","Data":"20505f0cb46feaf9546e851e545cc26885bb73078ee5bf6127ed84c9b198ae97"} Oct 30 06:32:20.362494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:20.362330 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20505f0cb46feaf9546e851e545cc26885bb73078ee5bf6127ed84c9b198ae97" Oct 30 06:32:20.363844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:20.363809 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" Oct 30 06:32:20.363844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:20.363833 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2" event={"ID":"6edc91af-0a89-46a6-bd4c-cbbb4aee2336","Type":"ContainerDied","Data":"1ad9ea0174e542b6c23b62d18b2e0550ef57069b9ff16a946f140a7990fbbec6"} Oct 30 06:32:20.364071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:20.363872 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ad9ea0174e542b6c23b62d18b2e0550ef57069b9ff16a946f140a7990fbbec6" Oct 30 06:32:20.647396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:20.638587 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:32:22.931929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:22.931880 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt"] Oct 30 06:32:22.932451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:22.932109 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" containerID="cri-o://329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3" gracePeriod=30 Oct 30 06:32:28.240981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.240955 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:32:28.306945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.306910 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-credential-socket\") pod \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " Oct 30 06:32:28.307175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.306979 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-kube-api-access-cn4mx\" (UniqueName: \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-kube-api-access-cn4mx\") pod \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " Oct 30 06:32:28.307175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.307052 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-podinfo\") pod \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " Oct 30 06:32:28.307175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.307164 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-credential-socket" pod "5923bc1d-e76f-44df-a3a9-6a3428c52d94" (UID: "5923bc1d-e76f-44df-a3a9-6a3428c52d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:28.307392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.307179 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-certs\") pod \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " Oct 30 06:32:28.307392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.307213 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-socket\") pod \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " Oct 30 06:32:28.307392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.307263 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istiod-ca-cert\") pod \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " Oct 30 06:32:28.307392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.307345 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-data\") pod \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " Oct 30 06:32:28.307627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.307395 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-token\" (UniqueName: \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-token\") pod \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " Oct 30 06:32:28.307627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.307425 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-envoy\") pod \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\" (UID: \"5923bc1d-e76f-44df-a3a9-6a3428c52d94\") " Oct 30 06:32:28.307627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.307462 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-certs" pod "5923bc1d-e76f-44df-a3a9-6a3428c52d94" (UID: "5923bc1d-e76f-44df-a3a9-6a3428c52d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:28.307790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.307652 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-data" pod "5923bc1d-e76f-44df-a3a9-6a3428c52d94" (UID: "5923bc1d-e76f-44df-a3a9-6a3428c52d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:28.307790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.307726 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-socket" pod "5923bc1d-e76f-44df-a3a9-6a3428c52d94" (UID: "5923bc1d-e76f-44df-a3a9-6a3428c52d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:28.308044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.308027 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:28.308108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.308046 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:28.308108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.308055 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:28.308108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.308064 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:28.308283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.308234 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istiod-ca-cert" pod "5923bc1d-e76f-44df-a3a9-6a3428c52d94" (UID: "5923bc1d-e76f-44df-a3a9-6a3428c52d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:32:28.310435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.310408 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-envoy" pod "5923bc1d-e76f-44df-a3a9-6a3428c52d94" (UID: "5923bc1d-e76f-44df-a3a9-6a3428c52d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:28.310687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.310659 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-podinfo" pod "5923bc1d-e76f-44df-a3a9-6a3428c52d94" (UID: "5923bc1d-e76f-44df-a3a9-6a3428c52d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:32:28.310954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.310933 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-token" pod "5923bc1d-e76f-44df-a3a9-6a3428c52d94" (UID: "5923bc1d-e76f-44df-a3a9-6a3428c52d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:28.311051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.310995 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-kube-api-access-cn4mx" pod "5923bc1d-e76f-44df-a3a9-6a3428c52d94" (UID: "5923bc1d-e76f-44df-a3a9-6a3428c52d94"). InnerVolumeSpecName "kube-api-access-cn4mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:28.409599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.409421 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:28.409599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.409462 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:28.409599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.409478 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:28.409599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.409494 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn4mx\" (UniqueName: \"kubernetes.io/projected/5923bc1d-e76f-44df-a3a9-6a3428c52d94-kube-api-access-cn4mx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:28.409599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.409509 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5923bc1d-e76f-44df-a3a9-6a3428c52d94-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:28.468669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.468578 5001 generic.go:356] "Generic (PLEG): container finished" podID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerID="329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3" exitCode=0 Oct 30 06:32:28.468850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.468820 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" event={"ID":"5923bc1d-e76f-44df-a3a9-6a3428c52d94","Type":"ContainerDied","Data":"329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3"} Oct 30 06:32:28.468917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.468897 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" event={"ID":"5923bc1d-e76f-44df-a3a9-6a3428c52d94","Type":"ContainerDied","Data":"461b2248c0d46fef1016043063c76afd05c3f98e87ac8c77fb5ca9a5c4064be6"} Oct 30 06:32:28.468974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.468934 5001 scope.go:122] "RemoveContainer" containerID="329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3" Oct 30 06:32:28.469219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.469171 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt" Oct 30 06:32:28.488730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.488705 5001 scope.go:122] "RemoveContainer" containerID="329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3" Oct 30 06:32:28.489055 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:32:28.489036 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3\": container with ID starting with 329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3 not found: ID does not exist" containerID="329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3" Oct 30 06:32:28.489150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.489063 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3"} err="failed to get container status \"329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3\": rpc error: code = NotFound desc = could not find container \"329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3\": container with ID starting with 329dfe743d512ba01872f1883ee0bb4f540ad322dba08ec895a2792f540396b3 not found: ID does not exist" Oct 30 06:32:28.530945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.530912 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt"] Oct 30 06:32:28.535154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:28.535125 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-b4hdt"] Oct 30 06:32:30.118890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:30.118847 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" path="/var/lib/kubelet/pods/5923bc1d-e76f-44df-a3a9-6a3428c52d94/volumes" Oct 30 06:32:32.510427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:32.510394 5001 generic.go:356] "Generic (PLEG): container finished" podID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerID="2e78a0807bb4d0fbc21fdf342360b4118d5881bbe47f5789e89cbb7d93d6bebe" exitCode=0 Oct 30 06:32:32.511002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:32.510477 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" event={"ID":"75ca4d25-1038-4aaa-9d74-436b3222223c","Type":"ContainerDied","Data":"2e78a0807bb4d0fbc21fdf342360b4118d5881bbe47f5789e89cbb7d93d6bebe"} Oct 30 06:32:32.511079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:32.511022 5001 scope.go:122] "RemoveContainer" containerID="2e78a0807bb4d0fbc21fdf342360b4118d5881bbe47f5789e89cbb7d93d6bebe" Oct 30 06:32:32.512706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:32.512665 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:32.512814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:32.512759 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:32.512814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:32.512795 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:33.519795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:33.519759 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" event={"ID":"75ca4d25-1038-4aaa-9d74-436b3222223c","Type":"ContainerStarted","Data":"f6080c7fb47873b68291a97b3faf495e9b3302bc6ea7a31767abc7392bc5af79"} Oct 30 06:32:34.461888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.461856 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:32:34.468466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.468427 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 30 06:32:34.468764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.468732 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 30 06:32:34.721958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.721799 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7318-f1b6c5"] Oct 30 06:32:34.726367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.723695 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" Oct 30 06:32:34.731394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.731365 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" Oct 30 06:32:34.731613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.731599 5001 cpu_manager.go:414] "containerMap: removing container" podUID="19764304-b00c-446e-9b9c-b3d30e8beebf" containerName="revoke-keys" Oct 30 06:32:34.756514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.756485 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19764304-b00c-446e-9b9c-b3d30e8beebf" containerName="revoke-keys" Oct 30 06:32:34.756801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.756787 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6edc91af-0a89-46a6-bd4c-cbbb4aee2336" containerName="revoke-keys" Oct 30 06:32:34.756885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.756875 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6edc91af-0a89-46a6-bd4c-cbbb4aee2336" containerName="revoke-keys" Oct 30 06:32:34.758330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.757248 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5923bc1d-e76f-44df-a3a9-6a3428c52d94" containerName="istio-proxy" Oct 30 06:32:34.758524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.758506 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="19764304-b00c-446e-9b9c-b3d30e8beebf" containerName="revoke-keys" Oct 30 06:32:34.758623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.758614 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6edc91af-0a89-46a6-bd4c-cbbb4aee2336" containerName="revoke-keys" Oct 30 06:32:34.781934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.781904 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-f1b6c5" Oct 30 06:32:34.807513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.782562 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7318-f1b6c5"] Oct 30 06:32:34.822600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.822560 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:32:34.842868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.842841 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p6zmk\"" Oct 30 06:32:34.887331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.855558 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 06:32:34.888447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.888413 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 06:32:34.893097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:34.889379 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4xp\" (UniqueName: \"kubernetes.io/projected/bd9ac01f-881a-447b-ac26-59f4961a1fe1-kube-api-access-tg4xp\") pod \"test-curl-7318-f1b6c5\" (UID: \"bd9ac01f-881a-447b-ac26-59f4961a1fe1\") " pod="opendatahub/test-curl-7318-f1b6c5" Oct 30 06:32:35.011790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:35.011686 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4xp\" (UniqueName: \"kubernetes.io/projected/bd9ac01f-881a-447b-ac26-59f4961a1fe1-kube-api-access-tg4xp\") pod \"test-curl-7318-f1b6c5\" (UID: \"bd9ac01f-881a-447b-ac26-59f4961a1fe1\") " pod="opendatahub/test-curl-7318-f1b6c5" Oct 30 06:32:35.035540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:35.035506 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4xp\" (UniqueName: \"kubernetes.io/projected/bd9ac01f-881a-447b-ac26-59f4961a1fe1-kube-api-access-tg4xp\") pod \"test-curl-7318-f1b6c5\" (UID: \"bd9ac01f-881a-447b-ac26-59f4961a1fe1\") " pod="opendatahub/test-curl-7318-f1b6c5" Oct 30 06:32:35.150755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:35.150717 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-f1b6c5" Oct 30 06:32:35.462333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:35.459695 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 30 06:32:35.462333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:35.459753 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 30 06:32:35.492423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:35.492380 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7318-f1b6c5"] Oct 30 06:32:35.494729 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:32:35.494685 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd9ac01f_881a_447b_ac26_59f4961a1fe1.slice/crio-38fa9e58eca19902ab32305fcd422325f01f090a1c52ffd276b279ffb660fce3 WatchSource:0}: Error finding container 38fa9e58eca19902ab32305fcd422325f01f090a1c52ffd276b279ffb660fce3: Status 404 returned error can't find the container with id 38fa9e58eca19902ab32305fcd422325f01f090a1c52ffd276b279ffb660fce3 Oct 30 06:32:35.541147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:35.541118 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7318-f1b6c5" event={"ID":"bd9ac01f-881a-447b-ac26-59f4961a1fe1","Type":"ContainerStarted","Data":"38fa9e58eca19902ab32305fcd422325f01f090a1c52ffd276b279ffb660fce3"} Oct 30 06:32:36.220765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.220712 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr"] Oct 30 06:32:36.224230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.224209 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.244639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.244608 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f0145f-openshift-default-dockercfg-dq4jw\"" Oct 30 06:32:36.300329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.299332 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr"] Oct 30 06:32:36.333951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.333908 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-istio-token\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.333951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.333959 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-envoy\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.334232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.334040 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-certs\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.334232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.334081 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-credential-socket\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.334232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.334143 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00363ae6-95cd-4372-9209-487f572af4a9-istiod-ca-cert\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.334232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.334163 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhscv\" (UniqueName: \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-kube-api-access-lhscv\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.334232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.334197 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-data\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.334232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.334217 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00363ae6-95cd-4372-9209-487f572af4a9-istio-podinfo\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.334466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.334262 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-socket\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.457334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.447291 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-certs\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.457334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.447401 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-credential-socket\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.457334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.447510 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00363ae6-95cd-4372-9209-487f572af4a9-istiod-ca-cert\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.457334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.447543 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhscv\" (UniqueName: \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-kube-api-access-lhscv\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.457334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.447599 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-data\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.457334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.447631 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00363ae6-95cd-4372-9209-487f572af4a9-istio-podinfo\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.457334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.447700 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-socket\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.457334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.447758 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-istio-token\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.457334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.447818 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-envoy\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.457334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.456132 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-certs\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.457334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.456419 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-credential-socket\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.458102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.457441 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-socket\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.458102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.457726 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-data\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.462327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.458348 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00363ae6-95cd-4372-9209-487f572af4a9-istiod-ca-cert\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.475376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.464400 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 30 06:32:36.475376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.464451 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 30 06:32:36.510331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.495689 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-envoy\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.531327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.514921 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00363ae6-95cd-4372-9209-487f572af4a9-istio-podinfo\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.535325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.532450 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhscv\" (UniqueName: \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-kube-api-access-lhscv\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.542331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.539730 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-istio-token\") pod \"e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.566605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.566070 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:36.649332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:36.640880 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7318-f1b6c5" event={"ID":"bd9ac01f-881a-447b-ac26-59f4961a1fe1","Type":"ContainerStarted","Data":"1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f"} Oct 30 06:32:37.084196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.084085 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7318-f1b6c5" podStartSLOduration=3.084063772 podStartE2EDuration="3.084063772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:36.71088348 +0000 UTC m=+2370.706840491" watchObservedRunningTime="2026-10-30 06:32:37.084063772 +0000 UTC m=+2371.080020784" Oct 30 06:32:37.094286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.094255 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr"] Oct 30 06:32:37.096953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.096913 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:37.097071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.096995 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:37.097071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.097042 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:37.459576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.459541 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 30 06:32:37.460266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.460218 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 30 06:32:37.666334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.659009 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" event={"ID":"00363ae6-95cd-4372-9209-487f572af4a9","Type":"ContainerStarted","Data":"7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d"} Oct 30 06:32:37.666334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.659069 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" event={"ID":"00363ae6-95cd-4372-9209-487f572af4a9","Type":"ContainerStarted","Data":"80a7ca8a876e760382fdbcb64d0c16ebd175060165c3731d9d1e489b16758a96"} Oct 30 06:32:37.704970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.704799 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7318-f1b6c5"] Oct 30 06:32:37.708826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.708794 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7318-f1b6c5"] Oct 30 06:32:37.715718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:37.714071 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" podStartSLOduration=1.714055325 podStartE2EDuration="1.714055325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:37.71171058 +0000 UTC m=+2371.707667592" watchObservedRunningTime="2026-10-30 06:32:37.714055325 +0000 UTC m=+2371.710012336" Oct 30 06:32:38.291932 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:32:38.291881 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:32:38.471334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:38.470905 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 30 06:32:38.471334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:38.470972 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 30 06:32:38.572511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:38.571389 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:38.584330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:38.583617 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:38.806155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:38.806111 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7318-f1b6c5" podUID="bd9ac01f-881a-447b-ac26-59f4961a1fe1" containerName="test-curl-7318-f1b6c5" containerID="cri-o://1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f" gracePeriod=2 Oct 30 06:32:38.807782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:38.807753 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:38.814193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:38.814162 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:32:39.621664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.621626 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g"] Oct 30 06:32:39.622871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.622805 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bd9ac01f-881a-447b-ac26-59f4961a1fe1" containerName="test-curl-7318-f1b6c5" Oct 30 06:32:39.622871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.622832 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9ac01f-881a-447b-ac26-59f4961a1fe1" containerName="test-curl-7318-f1b6c5" Oct 30 06:32:39.623132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.623125 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9ac01f-881a-447b-ac26-59f4961a1fe1" containerName="test-curl-7318-f1b6c5" Oct 30 06:32:39.627970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.627942 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.690761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.675926 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-69f3fa-openshift-default-dockercfg-ls84g\"" Oct 30 06:32:39.697217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.697183 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g"] Oct 30 06:32:39.714158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.714122 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.714419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.714196 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.714419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.714289 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.714567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.714440 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.714567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.714476 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.714567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.714504 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8rnv\" (UniqueName: \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-kube-api-access-w8rnv\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.714733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.714574 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.714733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.714622 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.714733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.714645 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.735355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.735217 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:32:39.737429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.737393 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:32:39.815739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.815700 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.815942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.815750 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.815942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.815780 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.815942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.815933 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.816102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.815992 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.816102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.816065 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.816224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.816199 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.816295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.816247 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.816427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.816199 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.816427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.816358 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.816548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.816476 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rnv\" (UniqueName: \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-kube-api-access-w8rnv\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.816613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.816580 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.816851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.816827 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.817111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.817094 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.818474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.818454 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.818703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.818686 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.839986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.839954 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.841581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.841553 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rnv\" (UniqueName: \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-kube-api-access-w8rnv\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:39.886411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.886296 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58"] Oct 30 06:32:39.892417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.892388 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:39.920712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.916677 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a0083e-openshift-default-dockercfg-zxj88\"" Oct 30 06:32:39.920712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.919250 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:39.920712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.919382 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:39.920712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.919427 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:39.920712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.919457 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfmj9\" (UniqueName: \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-kube-api-access-xfmj9\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:39.920712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.919501 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:39.920712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.919576 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:39.920712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.919651 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:39.920712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.919724 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:39.920712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.919749 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:39.923258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:39.923110 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58"] Oct 30 06:32:40.006579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.006547 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:40.021129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.021095 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.021257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.021213 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.021342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.021256 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.021342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.021284 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfmj9\" (UniqueName: \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-kube-api-access-xfmj9\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.021474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.021355 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.021474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.021430 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.021592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.021493 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.021592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.021546 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.021592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.021570 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.021787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.021757 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.022021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.021993 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.022140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.022050 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.022347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.022289 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.022443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.022367 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.024058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.024035 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.024157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.024054 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.139507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.120264 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfmj9\" (UniqueName: \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-kube-api-access-xfmj9\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.139507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.120797 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.229980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.229939 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:40.341138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.341073 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg"] Oct 30 06:32:40.357349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.351853 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.371410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.371356 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ad1661-openshift-default-dockercfg-2bbrc\"" Oct 30 06:32:40.383466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.383430 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg"] Oct 30 06:32:40.411787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.411694 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.411787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.411739 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.411787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.411762 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.412030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.411805 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4r65\" (UniqueName: \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-kube-api-access-s4r65\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.412030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.411853 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.412030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.411870 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.412030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.411914 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.412030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.411997 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.412200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.412049 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.566170 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.566280 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.566341 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.566376 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.566445 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4r65\" (UniqueName: \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-kube-api-access-s4r65\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.566524 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.566551 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.566604 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.566718 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.568046 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.568768 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.569808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.569671 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.575505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.569923 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.575505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.570491 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.575505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.572609 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.575505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.574871 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.600647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.600620 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4r65\" (UniqueName: \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-kube-api-access-s4r65\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.612329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.611238 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.619107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.617429 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g"] Oct 30 06:32:40.648334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.625892 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:40.648334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.627227 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:40.648334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.627340 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:40.768573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.754188 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:40.882448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.871553 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58"] Oct 30 06:32:40.950397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.950363 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" event={"ID":"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78","Type":"ContainerStarted","Data":"38ba3fd6feb388d8dbb2f71d5983d0dc37aa246ad731f909c64a83366430f82d"} Oct 30 06:32:40.972090 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:32:40.972062 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdd5e60_c7ac_460d_883a_f4e4961f3735.slice/crio-1b42ce9bf182efafb3412b2b222733a0f73741295208c14dee01f56014bcb8ae WatchSource:0}: Error finding container 1b42ce9bf182efafb3412b2b222733a0f73741295208c14dee01f56014bcb8ae: Status 404 returned error can't find the container with id 1b42ce9bf182efafb3412b2b222733a0f73741295208c14dee01f56014bcb8ae Oct 30 06:32:40.980185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.980134 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:40.980352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.980256 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:40.980352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:40.980320 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:41.237761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.237732 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-f1b6c5" Oct 30 06:32:41.315969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.315850 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9ac01f-881a-447b-ac26-59f4961a1fe1-kube-api-access-tg4xp\" (UniqueName: \"kubernetes.io/projected/bd9ac01f-881a-447b-ac26-59f4961a1fe1-kube-api-access-tg4xp\") pod \"bd9ac01f-881a-447b-ac26-59f4961a1fe1\" (UID: \"bd9ac01f-881a-447b-ac26-59f4961a1fe1\") " Oct 30 06:32:41.318697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.318565 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9ac01f-881a-447b-ac26-59f4961a1fe1-kube-api-access-tg4xp" pod "bd9ac01f-881a-447b-ac26-59f4961a1fe1" (UID: "bd9ac01f-881a-447b-ac26-59f4961a1fe1"). InnerVolumeSpecName "kube-api-access-tg4xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:41.322210 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:32:41.322185 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bf08e0_cac0_43ed_847a_3df6d9003e3c.slice/crio-2b2c970903384201e91e1e1860ad331f9b680d8c33dab3aa025cf5de4b761437 WatchSource:0}: Error finding container 2b2c970903384201e91e1e1860ad331f9b680d8c33dab3aa025cf5de4b761437: Status 404 returned error can't find the container with id 2b2c970903384201e91e1e1860ad331f9b680d8c33dab3aa025cf5de4b761437 Oct 30 06:32:41.322538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.322515 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg"] Oct 30 06:32:41.325575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.325533 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:41.325678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.325642 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:41.325726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.325707 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:41.418395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.418360 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg4xp\" (UniqueName: \"kubernetes.io/projected/bd9ac01f-881a-447b-ac26-59f4961a1fe1-kube-api-access-tg4xp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:41.962340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.962272 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" event={"ID":"7fdd5e60-c7ac-460d-883a-f4e4961f3735","Type":"ContainerStarted","Data":"29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a"} Oct 30 06:32:41.962793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.962379 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" event={"ID":"7fdd5e60-c7ac-460d-883a-f4e4961f3735","Type":"ContainerStarted","Data":"1b42ce9bf182efafb3412b2b222733a0f73741295208c14dee01f56014bcb8ae"} Oct 30 06:32:41.964465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.964433 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" event={"ID":"b2bf08e0-cac0-43ed-847a-3df6d9003e3c","Type":"ContainerStarted","Data":"5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f"} Oct 30 06:32:41.964607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.964486 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" event={"ID":"b2bf08e0-cac0-43ed-847a-3df6d9003e3c","Type":"ContainerStarted","Data":"2b2c970903384201e91e1e1860ad331f9b680d8c33dab3aa025cf5de4b761437"} Oct 30 06:32:41.966623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.966597 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" event={"ID":"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78","Type":"ContainerStarted","Data":"d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c"} Oct 30 06:32:41.968004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.967977 5001 generic.go:356] "Generic (PLEG): container finished" podID="bd9ac01f-881a-447b-ac26-59f4961a1fe1" containerID="1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f" exitCode=137 Oct 30 06:32:41.968110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.968020 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-f1b6c5" Oct 30 06:32:41.968110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.968029 5001 scope.go:122] "RemoveContainer" containerID="1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f" Oct 30 06:32:41.980102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.980075 5001 scope.go:122] "RemoveContainer" containerID="1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f" Oct 30 06:32:41.980435 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:32:41.980414 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f\": container with ID starting with 1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f not found: ID does not exist" containerID="1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f" Oct 30 06:32:41.980536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:41.980441 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f"} err="failed to get container status \"1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f\": rpc error: code = NotFound desc = could not find container \"1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f\": container with ID starting with 1cfe56526f0e7e12fc979fa9bb3f010d3456ee6003c18e56b9437f494bd3611f not found: ID does not exist" Oct 30 06:32:42.006886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.006849 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:42.010217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.010168 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" podStartSLOduration=3.010153258 podStartE2EDuration="3.010153258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:42.006424329 +0000 UTC m=+2376.002381330" watchObservedRunningTime="2026-10-30 06:32:42.010153258 +0000 UTC m=+2376.006110267" Oct 30 06:32:42.013285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.013262 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:42.053491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.053434 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" podStartSLOduration=3.053414702 podStartE2EDuration="3.053414702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:42.049811288 +0000 UTC m=+2376.045768295" watchObservedRunningTime="2026-10-30 06:32:42.053414702 +0000 UTC m=+2376.049371712" Oct 30 06:32:42.062546 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:32:42.062509 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7318-f1b6c5\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" podUID="bd9ac01f-881a-447b-ac26-59f4961a1fe1" pod="opendatahub/test-curl-7318-f1b6c5" Oct 30 06:32:42.108531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.108479 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" podStartSLOduration=2.108461307 podStartE2EDuration="2.108461307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:42.105361686 +0000 UTC m=+2376.101318694" watchObservedRunningTime="2026-10-30 06:32:42.108461307 +0000 UTC m=+2376.104418310" Oct 30 06:32:42.119569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.119539 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9ac01f-881a-447b-ac26-59f4961a1fe1" path="/var/lib/kubelet/pods/bd9ac01f-881a-447b-ac26-59f4961a1fe1/volumes" Oct 30 06:32:42.169790 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:32:42.169748 5001 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7318-f1b6c5\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" podUID="bd9ac01f-881a-447b-ac26-59f4961a1fe1" pod="opendatahub/test-curl-7318-f1b6c5" Oct 30 06:32:42.231057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.231010 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:42.238606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.238571 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:42.414806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.414728 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x"] Oct 30 06:32:42.419338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.419294 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:32:42.436875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.436842 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f0145f\"" Oct 30 06:32:42.437520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.437497 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f0145f-dockercfg-9bm7b\"" Oct 30 06:32:42.440200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.440180 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x"] Oct 30 06:32:42.554371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.553663 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ede9e-3c58-48c1-a663-1299268d6838-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x\" (UID: \"e15ede9e-3c58-48c1-a663-1299268d6838\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:32:42.554371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.553747 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpmx\" (UniqueName: \"kubernetes.io/projected/e15ede9e-3c58-48c1-a663-1299268d6838-kube-api-access-mtpmx\") pod \"payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x\" (UID: \"e15ede9e-3c58-48c1-a663-1299268d6838\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:32:42.584330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.579940 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4"] Oct 30 06:32:42.713715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.704127 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpmx\" (UniqueName: \"kubernetes.io/projected/e15ede9e-3c58-48c1-a663-1299268d6838-kube-api-access-mtpmx\") pod \"payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x\" (UID: \"e15ede9e-3c58-48c1-a663-1299268d6838\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:32:42.713715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.704605 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ede9e-3c58-48c1-a663-1299268d6838-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x\" (UID: \"e15ede9e-3c58-48c1-a663-1299268d6838\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:32:42.762090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.747460 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ede9e-3c58-48c1-a663-1299268d6838-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x\" (UID: \"e15ede9e-3c58-48c1-a663-1299268d6838\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:32:42.815574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.815545 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:42.815871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.815737 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:32:42.816055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.816043 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:42.816151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.816142 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4"] Oct 30 06:32:42.826504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.826474 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpmx\" (UniqueName: \"kubernetes.io/projected/e15ede9e-3c58-48c1-a663-1299268d6838-kube-api-access-mtpmx\") pod \"payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x\" (UID: \"e15ede9e-3c58-48c1-a663-1299268d6838\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:32:42.979834 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.979746 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:42.980407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.980389 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:42.980566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.980549 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:42.980657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.980615 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:32:42.980881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.980848 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:32:42.980940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:42.980902 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:32:43.032939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.032903 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:32:43.034758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.034726 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5p74\" (UniqueName: \"kubernetes.io/projected/c553d6a6-04ad-43a5-abbd-73889c0797bb-kube-api-access-j5p74\") pod \"payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4\" (UID: \"c553d6a6-04ad-43a5-abbd-73889c0797bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:32:43.036654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.035243 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c553d6a6-04ad-43a5-abbd-73889c0797bb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4\" (UID: \"c553d6a6-04ad-43a5-abbd-73889c0797bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:32:43.138075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.136585 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c553d6a6-04ad-43a5-abbd-73889c0797bb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4\" (UID: \"c553d6a6-04ad-43a5-abbd-73889c0797bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:32:43.138075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.136710 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5p74\" (UniqueName: \"kubernetes.io/projected/c553d6a6-04ad-43a5-abbd-73889c0797bb-kube-api-access-j5p74\") pod \"payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4\" (UID: \"c553d6a6-04ad-43a5-abbd-73889c0797bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:32:43.138354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.138245 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c553d6a6-04ad-43a5-abbd-73889c0797bb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4\" (UID: \"c553d6a6-04ad-43a5-abbd-73889c0797bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:32:43.162898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.162686 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5p74\" (UniqueName: \"kubernetes.io/projected/c553d6a6-04ad-43a5-abbd-73889c0797bb-kube-api-access-j5p74\") pod \"payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4\" (UID: \"c553d6a6-04ad-43a5-abbd-73889c0797bb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:32:43.386014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.385978 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x"] Oct 30 06:32:43.392970 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:32:43.391975 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15ede9e_3c58_48c1_a663_1299268d6838.slice/crio-ae6fbac0b65a327f75c340c51af4126505f2cd2239551560ccc9054e86945c9e WatchSource:0}: Error finding container ae6fbac0b65a327f75c340c51af4126505f2cd2239551560ccc9054e86945c9e: Status 404 returned error can't find the container with id ae6fbac0b65a327f75c340c51af4126505f2cd2239551560ccc9054e86945c9e Oct 30 06:32:43.459178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.459148 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:32:43.827995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.827968 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4"] Oct 30 06:32:43.830083 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:32:43.830047 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc553d6a6_04ad_43a5_abbd_73889c0797bb.slice/crio-0561db3827a409a8f41a9a089d50507d1aa328abdafe553bfc434262cad987e9 WatchSource:0}: Error finding container 0561db3827a409a8f41a9a089d50507d1aa328abdafe553bfc434262cad987e9: Status 404 returned error can't find the container with id 0561db3827a409a8f41a9a089d50507d1aa328abdafe553bfc434262cad987e9 Oct 30 06:32:43.991589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.991546 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" event={"ID":"c553d6a6-04ad-43a5-abbd-73889c0797bb","Type":"ContainerStarted","Data":"01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e"} Oct 30 06:32:43.992085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.991624 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" event={"ID":"c553d6a6-04ad-43a5-abbd-73889c0797bb","Type":"ContainerStarted","Data":"0561db3827a409a8f41a9a089d50507d1aa328abdafe553bfc434262cad987e9"} Oct 30 06:32:43.992085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.991868 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:32:43.994937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.994902 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" event={"ID":"e15ede9e-3c58-48c1-a663-1299268d6838","Type":"ContainerStarted","Data":"db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164"} Oct 30 06:32:43.995089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.994964 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:32:43.995089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:43.994990 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" event={"ID":"e15ede9e-3c58-48c1-a663-1299268d6838","Type":"ContainerStarted","Data":"ae6fbac0b65a327f75c340c51af4126505f2cd2239551560ccc9054e86945c9e"} Oct 30 06:32:44.033106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:44.033046 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" podStartSLOduration=2.033031846 podStartE2EDuration="2.033031846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:44.030491195 +0000 UTC m=+2378.026448204" watchObservedRunningTime="2026-10-30 06:32:44.033031846 +0000 UTC m=+2378.028988856" Oct 30 06:32:44.076111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:44.076000 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" podStartSLOduration=2.075979647 podStartE2EDuration="2.075979647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:44.069797618 +0000 UTC m=+2378.065754629" watchObservedRunningTime="2026-10-30 06:32:44.075979647 +0000 UTC m=+2378.071936662" Oct 30 06:32:51.095923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.095830 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk"] Oct 30 06:32:51.101705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.101685 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:51.119751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.119724 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk"] Oct 30 06:32:51.126159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.126129 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:32:51.126344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.126190 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:32:51.126344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.126203 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:32:51.132241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.132218 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:32:51.196768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.196730 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c332a5-rcshk\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:51.197010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.196810 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c332a5-rcshk\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:51.197117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.197085 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhhf\" (UniqueName: \"kubernetes.io/projected/434c741e-6f54-4d58-ac23-66ab4663e540-kube-api-access-qjhhf\") pod \"maas-api-revoke-sub-71c332a5-rcshk\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:51.298864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.298829 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c332a5-rcshk\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:51.299088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.298889 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c332a5-rcshk\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:51.299088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.299014 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhhf\" (UniqueName: \"kubernetes.io/projected/434c741e-6f54-4d58-ac23-66ab4663e540-kube-api-access-qjhhf\") pod \"maas-api-revoke-sub-71c332a5-rcshk\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:51.299686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.299662 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c332a5-rcshk\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:51.301770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.301744 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c332a5-rcshk\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:51.324071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.324041 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhhf\" (UniqueName: \"kubernetes.io/projected/434c741e-6f54-4d58-ac23-66ab4663e540-kube-api-access-qjhhf\") pod \"maas-api-revoke-sub-71c332a5-rcshk\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:51.416566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.416532 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:51.794881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:51.794855 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk"] Oct 30 06:32:51.797665 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:32:51.797630 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434c741e_6f54_4d58_ac23_66ab4663e540.slice/crio-a22bd92de0fc36e7735a0205425bebfa12afc43c0028c60c9fd094cb2b688e81 WatchSource:0}: Error finding container a22bd92de0fc36e7735a0205425bebfa12afc43c0028c60c9fd094cb2b688e81: Status 404 returned error can't find the container with id a22bd92de0fc36e7735a0205425bebfa12afc43c0028c60c9fd094cb2b688e81 Oct 30 06:32:52.108368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:52.108335 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" event={"ID":"434c741e-6f54-4d58-ac23-66ab4663e540","Type":"ContainerStarted","Data":"a22bd92de0fc36e7735a0205425bebfa12afc43c0028c60c9fd094cb2b688e81"} Oct 30 06:32:53.118575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:53.118543 5001 generic.go:356] "Generic (PLEG): container finished" podID="434c741e-6f54-4d58-ac23-66ab4663e540" containerID="433a777e81af1df4bf0596728995511772638c325f421913fb961a2dcf823fa7" exitCode=0 Oct 30 06:32:53.119001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:53.118629 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" event={"ID":"434c741e-6f54-4d58-ac23-66ab4663e540","Type":"ContainerDied","Data":"433a777e81af1df4bf0596728995511772638c325f421913fb961a2dcf823fa7"} Oct 30 06:32:54.304700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:54.304666 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:54.455772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:54.455678 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-xks-ca\") pod \"434c741e-6f54-4d58-ac23-66ab4663e540\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " Oct 30 06:32:54.455971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:54.455874 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434c741e-6f54-4d58-ac23-66ab4663e540-kube-api-access-qjhhf\" (UniqueName: \"kubernetes.io/projected/434c741e-6f54-4d58-ac23-66ab4663e540-kube-api-access-qjhhf\") pod \"434c741e-6f54-4d58-ac23-66ab4663e540\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " Oct 30 06:32:54.455971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:54.455936 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-openshift-ca\") pod \"434c741e-6f54-4d58-ac23-66ab4663e540\" (UID: \"434c741e-6f54-4d58-ac23-66ab4663e540\") " Oct 30 06:32:54.456350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:54.456276 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-openshift-ca" pod "434c741e-6f54-4d58-ac23-66ab4663e540" (UID: "434c741e-6f54-4d58-ac23-66ab4663e540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:32:54.456746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:54.456722 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:54.458061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:54.458034 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-xks-ca" pod "434c741e-6f54-4d58-ac23-66ab4663e540" (UID: "434c741e-6f54-4d58-ac23-66ab4663e540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:32:54.458363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:54.458332 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434c741e-6f54-4d58-ac23-66ab4663e540-kube-api-access-qjhhf" pod "434c741e-6f54-4d58-ac23-66ab4663e540" (UID: "434c741e-6f54-4d58-ac23-66ab4663e540"). InnerVolumeSpecName "kube-api-access-qjhhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:54.558202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:54.558169 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhhf\" (UniqueName: \"kubernetes.io/projected/434c741e-6f54-4d58-ac23-66ab4663e540-kube-api-access-qjhhf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:54.558202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:54.558199 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434c741e-6f54-4d58-ac23-66ab4663e540-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:32:55.008966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:55.008939 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:32:55.009149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:55.009013 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:32:55.160659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:55.160619 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" event={"ID":"434c741e-6f54-4d58-ac23-66ab4663e540","Type":"ContainerDied","Data":"a22bd92de0fc36e7735a0205425bebfa12afc43c0028c60c9fd094cb2b688e81"} Oct 30 06:32:55.160868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:55.160685 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a22bd92de0fc36e7735a0205425bebfa12afc43c0028c60c9fd094cb2b688e81" Oct 30 06:32:55.160868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:55.160758 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk" Oct 30 06:32:59.936690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:59.936656 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk"] Oct 30 06:32:59.937162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:32:59.936856 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" containerID="cri-o://36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e" gracePeriod=30 Oct 30 06:33:01.525126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.516616 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt"] Oct 30 06:33:01.525126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.518219 5001 cpu_manager.go:414] "containerMap: removing container" podUID="434c741e-6f54-4d58-ac23-66ab4663e540" containerName="revoke-keys" Oct 30 06:33:01.525126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.518248 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434c741e-6f54-4d58-ac23-66ab4663e540" containerName="revoke-keys" Oct 30 06:33:01.525126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.518660 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="434c741e-6f54-4d58-ac23-66ab4663e540" containerName="revoke-keys" Oct 30 06:33:01.526022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.525677 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:01.541632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.541604 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt"] Oct 30 06:33:01.546598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.546570 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:33:01.546840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.546625 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:33:01.571136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.571102 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:33:01.571384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.571165 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:33:01.622833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.622798 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d78318cf-hvzbt\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:01.623042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.622962 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwnl\" (UniqueName: \"kubernetes.io/projected/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-kube-api-access-cnwnl\") pod \"maas-api-revoke-sub-d78318cf-hvzbt\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:01.623101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.623052 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d78318cf-hvzbt\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:01.724908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.724865 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d78318cf-hvzbt\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:01.725116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.724977 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwnl\" (UniqueName: \"kubernetes.io/projected/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-kube-api-access-cnwnl\") pod \"maas-api-revoke-sub-d78318cf-hvzbt\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:01.725116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.725022 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d78318cf-hvzbt\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:01.725676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.725651 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d78318cf-hvzbt\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:01.727864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.727835 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d78318cf-hvzbt\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:01.770326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.770275 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwnl\" (UniqueName: \"kubernetes.io/projected/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-kube-api-access-cnwnl\") pod \"maas-api-revoke-sub-d78318cf-hvzbt\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:01.843441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:01.843333 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:02.156439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:02.156411 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt"] Oct 30 06:33:02.158274 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:33:02.158253 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac6147c_2278_4bc2_9a3a_12fecfdd4ae0.slice/crio-cec992f545aede0451be4be66508bba2aab328b615c06fef70e48bf154d67fa0 WatchSource:0}: Error finding container cec992f545aede0451be4be66508bba2aab328b615c06fef70e48bf154d67fa0: Status 404 returned error can't find the container with id cec992f545aede0451be4be66508bba2aab328b615c06fef70e48bf154d67fa0 Oct 30 06:33:02.224080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:02.224042 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" event={"ID":"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0","Type":"ContainerStarted","Data":"cec992f545aede0451be4be66508bba2aab328b615c06fef70e48bf154d67fa0"} Oct 30 06:33:03.235429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:03.235387 5001 generic.go:356] "Generic (PLEG): container finished" podID="dac6147c-2278-4bc2-9a3a-12fecfdd4ae0" containerID="9c82b9591544b4044141d85e0e20dc6ce4e6ca86fb3033f82fd707aa3641cc89" exitCode=0 Oct 30 06:33:03.235875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:03.235475 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" event={"ID":"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0","Type":"ContainerDied","Data":"9c82b9591544b4044141d85e0e20dc6ce4e6ca86fb3033f82fd707aa3641cc89"} Oct 30 06:33:04.426847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:04.426820 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:04.584372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:04.584261 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-kube-api-access-cnwnl\" (UniqueName: \"kubernetes.io/projected/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-kube-api-access-cnwnl\") pod \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " Oct 30 06:33:04.584546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:04.584403 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-openshift-ca\") pod \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " Oct 30 06:33:04.584624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:04.584558 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-xks-ca\") pod \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\" (UID: \"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0\") " Oct 30 06:33:04.584841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:04.584813 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-openshift-ca" pod "dac6147c-2278-4bc2-9a3a-12fecfdd4ae0" (UID: "dac6147c-2278-4bc2-9a3a-12fecfdd4ae0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:33:04.585138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:04.585118 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:04.586769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:04.586739 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-kube-api-access-cnwnl" pod "dac6147c-2278-4bc2-9a3a-12fecfdd4ae0" (UID: "dac6147c-2278-4bc2-9a3a-12fecfdd4ae0"). InnerVolumeSpecName "kube-api-access-cnwnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:04.586916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:04.586851 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-xks-ca" pod "dac6147c-2278-4bc2-9a3a-12fecfdd4ae0" (UID: "dac6147c-2278-4bc2-9a3a-12fecfdd4ae0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:33:04.686821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:04.686781 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnwnl\" (UniqueName: \"kubernetes.io/projected/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-kube-api-access-cnwnl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:04.686821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:04.686816 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:05.217717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.217655 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:33:05.252583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.252547 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" event={"ID":"dac6147c-2278-4bc2-9a3a-12fecfdd4ae0","Type":"ContainerDied","Data":"cec992f545aede0451be4be66508bba2aab328b615c06fef70e48bf154d67fa0"} Oct 30 06:33:05.252794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.252603 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cec992f545aede0451be4be66508bba2aab328b615c06fef70e48bf154d67fa0" Oct 30 06:33:05.252794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.252557 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt" Oct 30 06:33:05.254388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.254354 5001 generic.go:356] "Generic (PLEG): container finished" podID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerID="36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e" exitCode=0 Oct 30 06:33:05.254535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.254424 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" Oct 30 06:33:05.254535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.254479 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" event={"ID":"d343287d-c9ed-4e9b-9891-438d1007cccb","Type":"ContainerDied","Data":"36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e"} Oct 30 06:33:05.254684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.254537 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk" event={"ID":"d343287d-c9ed-4e9b-9891-438d1007cccb","Type":"ContainerDied","Data":"534dcccbb140f7fa36bd7a738e7c8af999fbc58e2305c37293508e370149b211"} Oct 30 06:33:05.254684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.254564 5001 scope.go:122] "RemoveContainer" containerID="36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e" Oct 30 06:33:05.280284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.280251 5001 scope.go:122] "RemoveContainer" containerID="36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e" Oct 30 06:33:05.280850 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:33:05.280814 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e\": container with ID starting with 36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e not found: ID does not exist" containerID="36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e" Oct 30 06:33:05.280988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.280857 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e"} err="failed to get container status \"36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e\": rpc error: code = NotFound desc = could not find container \"36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e\": container with ID starting with 36a65a54836c8d72559d0432dc42746f44ff55def11410fd057cfcfdef32b08e not found: ID does not exist" Oct 30 06:33:05.295647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.295618 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-podinfo\") pod \"d343287d-c9ed-4e9b-9891-438d1007cccb\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " Oct 30 06:33:05.295875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.295855 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-kube-api-access-bpfx4\" (UniqueName: \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-kube-api-access-bpfx4\") pod \"d343287d-c9ed-4e9b-9891-438d1007cccb\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " Oct 30 06:33:05.295973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.295953 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-envoy\") pod \"d343287d-c9ed-4e9b-9891-438d1007cccb\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " Oct 30 06:33:05.296049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.296017 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-certs\") pod \"d343287d-c9ed-4e9b-9891-438d1007cccb\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " Oct 30 06:33:05.296350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.296286 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-certs" pod "d343287d-c9ed-4e9b-9891-438d1007cccb" (UID: "d343287d-c9ed-4e9b-9891-438d1007cccb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:05.296454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.296360 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-token\" (UniqueName: \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-token\") pod \"d343287d-c9ed-4e9b-9891-438d1007cccb\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " Oct 30 06:33:05.297458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.296544 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d343287d-c9ed-4e9b-9891-438d1007cccb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d343287d-c9ed-4e9b-9891-438d1007cccb-istiod-ca-cert\") pod \"d343287d-c9ed-4e9b-9891-438d1007cccb\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " Oct 30 06:33:05.297458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.296592 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-data\") pod \"d343287d-c9ed-4e9b-9891-438d1007cccb\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " Oct 30 06:33:05.297458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.296641 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-credential-socket\") pod \"d343287d-c9ed-4e9b-9891-438d1007cccb\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " Oct 30 06:33:05.297458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.296670 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-socket\") pod \"d343287d-c9ed-4e9b-9891-438d1007cccb\" (UID: \"d343287d-c9ed-4e9b-9891-438d1007cccb\") " Oct 30 06:33:05.297458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.296897 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-data" pod "d343287d-c9ed-4e9b-9891-438d1007cccb" (UID: "d343287d-c9ed-4e9b-9891-438d1007cccb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:05.297458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.297251 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-credential-socket" pod "d343287d-c9ed-4e9b-9891-438d1007cccb" (UID: "d343287d-c9ed-4e9b-9891-438d1007cccb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:05.297897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.297474 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-socket" pod "d343287d-c9ed-4e9b-9891-438d1007cccb" (UID: "d343287d-c9ed-4e9b-9891-438d1007cccb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:05.297897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.297589 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:05.297897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.297614 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:05.297897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.297629 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:05.298115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.297927 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d343287d-c9ed-4e9b-9891-438d1007cccb-istiod-ca-cert" pod "d343287d-c9ed-4e9b-9891-438d1007cccb" (UID: "d343287d-c9ed-4e9b-9891-438d1007cccb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:33:05.299807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.299780 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-envoy" pod "d343287d-c9ed-4e9b-9891-438d1007cccb" (UID: "d343287d-c9ed-4e9b-9891-438d1007cccb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:05.300299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.300269 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-token" pod "d343287d-c9ed-4e9b-9891-438d1007cccb" (UID: "d343287d-c9ed-4e9b-9891-438d1007cccb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:05.301029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.301003 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-podinfo" pod "d343287d-c9ed-4e9b-9891-438d1007cccb" (UID: "d343287d-c9ed-4e9b-9891-438d1007cccb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:33:05.301282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.301257 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-kube-api-access-bpfx4" pod "d343287d-c9ed-4e9b-9891-438d1007cccb" (UID: "d343287d-c9ed-4e9b-9891-438d1007cccb"). InnerVolumeSpecName "kube-api-access-bpfx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:05.398909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.398873 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:05.398909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.398911 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpfx4\" (UniqueName: \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-kube-api-access-bpfx4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:05.399142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.398925 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:05.399142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.398938 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d343287d-c9ed-4e9b-9891-438d1007cccb-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:05.399142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.398950 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d343287d-c9ed-4e9b-9891-438d1007cccb-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:05.399142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.398963 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d343287d-c9ed-4e9b-9891-438d1007cccb-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:05.602640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.602603 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk"] Oct 30 06:33:05.607098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:05.607069 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6g2sk"] Oct 30 06:33:06.120155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:06.120124 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" path="/var/lib/kubelet/pods/d343287d-c9ed-4e9b-9891-438d1007cccb/volumes" Oct 30 06:33:06.792380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:06.792344 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:33:06.808426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:06.808403 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:33:08.266744 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:33:08.266699 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:33:09.949546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:09.949513 5001 scope.go:122] "RemoveContainer" containerID="5d940b09b950eca0697ad2acc8c26e19e063fa94edbe23bc282149d0ae595d63" Oct 30 06:33:09.963502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:09.963374 5001 scope.go:122] "RemoveContainer" containerID="59e99048e70f707cbe16c44b9dd729030ae6014ba5db0e58cdff34bfab19ab7f" Oct 30 06:33:09.978181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:09.978160 5001 scope.go:122] "RemoveContainer" containerID="5e667b10e6fa190738fad88796fd76daae0ab594a471be5752124129a06b9205" Oct 30 06:33:09.990714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:09.990682 5001 scope.go:122] "RemoveContainer" containerID="c26cc63e7b03fce5eb3107bf73beb9b656f3a52d174f0afd851ac86d0fb473c4" Oct 30 06:33:10.001561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:10.001536 5001 scope.go:122] "RemoveContainer" containerID="720e17bca026c9c3ed536effcab6f94159f2bad12fa10d7604a36662aaeca6ef" Oct 30 06:33:12.874763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:12.874728 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:33:28.481888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:28.481851 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht"] Oct 30 06:33:30.011686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.011638 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj"] Oct 30 06:33:30.012879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.012849 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" Oct 30 06:33:30.012879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.012876 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" Oct 30 06:33:30.013144 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.012918 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dac6147c-2278-4bc2-9a3a-12fecfdd4ae0" containerName="revoke-keys" Oct 30 06:33:30.013144 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.012926 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac6147c-2278-4bc2-9a3a-12fecfdd4ae0" containerName="revoke-keys" Oct 30 06:33:30.013144 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.013057 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d343287d-c9ed-4e9b-9891-438d1007cccb" containerName="istio-proxy" Oct 30 06:33:30.013144 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.013073 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac6147c-2278-4bc2-9a3a-12fecfdd4ae0" containerName="revoke-keys" Oct 30 06:33:30.020078 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.020052 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.026116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.026083 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.026247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.026157 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.026247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.026210 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.026247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.026233 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qrz\" (UniqueName: \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-kube-api-access-m4qrz\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.026410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.026262 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.026410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.026286 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.026410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.026327 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.026410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.026384 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.026705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.026411 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.032354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.030590 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj"] Oct 30 06:33:30.035050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.035019 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a88d7d-openshift-default-dockercfg-s6dkq\"" Oct 30 06:33:30.174790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.174756 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.175385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.175361 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.176135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.176117 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.176831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.176647 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.177071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.177050 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.177630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.177612 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.178361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.178058 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qrz\" (UniqueName: \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-kube-api-access-m4qrz\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.178572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.178553 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.189854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.177500 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.189854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.177990 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.189854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.179281 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.190486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.190427 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.190636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.189744 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.190787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.190773 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.201565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.201513 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.236868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.236837 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qrz\" (UniqueName: \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-kube-api-access-m4qrz\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.272212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.272099 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.280397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.273205 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.344991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.344954 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:30.739110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.739077 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj"] Oct 30 06:33:30.740615 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:33:30.740585 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053335c2_28a1_476e_b3d8_9d8bf3c5e005.slice/crio-36a46a213c154098df882e8d83010a2408940da60b0e2dfa74c3b9fd3b84aae5 WatchSource:0}: Error finding container 36a46a213c154098df882e8d83010a2408940da60b0e2dfa74c3b9fd3b84aae5: Status 404 returned error can't find the container with id 36a46a213c154098df882e8d83010a2408940da60b0e2dfa74c3b9fd3b84aae5 Oct 30 06:33:30.743534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.743500 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:30.743620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.743572 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:30.743620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.743602 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:30.755392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:30.755347 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" event={"ID":"053335c2-28a1-476e-b3d8-9d8bf3c5e005","Type":"ContainerStarted","Data":"36a46a213c154098df882e8d83010a2408940da60b0e2dfa74c3b9fd3b84aae5"} Oct 30 06:33:31.768525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:31.768485 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" event={"ID":"053335c2-28a1-476e-b3d8-9d8bf3c5e005","Type":"ContainerStarted","Data":"fc0690a4ef141685d5cfde802fa036ac592cedb4dbd57ef0c6085aea73046a5f"} Oct 30 06:33:31.836132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:31.836081 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" podStartSLOduration=2.8360664719999997 podStartE2EDuration="2.836066472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:31.823876658 +0000 UTC m=+2425.819833696" watchObservedRunningTime="2026-10-30 06:33:31.836066472 +0000 UTC m=+2425.832023479" Oct 30 06:33:32.356332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:32.353642 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:32.550347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:32.547635 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:32.777745 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:32.777660 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:32.779009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:32.778981 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:33:38.550331 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:33:38.545553 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:33:43.514010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:43.513937 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" podUID="84a96c90-0ab3-4482-91b9-cb9282434c1c" containerName="main" containerID="cri-o://8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972" gracePeriod=45 Oct 30 06:33:43.954179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:43.954151 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:33:44.022850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.022499 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:33:44.022850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.022591 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:33:44.025452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.025416 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:33:44.025600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.025486 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:33:44.025808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.025776 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-model-cache\") pod \"84a96c90-0ab3-4482-91b9-cb9282434c1c\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " Oct 30 06:33:44.025974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.025955 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a96c90-0ab3-4482-91b9-cb9282434c1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/84a96c90-0ab3-4482-91b9-cb9282434c1c-tls-certs\") pod \"84a96c90-0ab3-4482-91b9-cb9282434c1c\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " Oct 30 06:33:44.026053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.026024 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-tmp-dir\") pod \"84a96c90-0ab3-4482-91b9-cb9282434c1c\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " Oct 30 06:33:44.026118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.026104 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-home\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-home\") pod \"84a96c90-0ab3-4482-91b9-cb9282434c1c\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " Oct 30 06:33:44.026178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.026144 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-dshm\") pod \"84a96c90-0ab3-4482-91b9-cb9282434c1c\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " Oct 30 06:33:44.026247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.026184 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a96c90-0ab3-4482-91b9-cb9282434c1c-kube-api-access-klmml\" (UniqueName: \"kubernetes.io/projected/84a96c90-0ab3-4482-91b9-cb9282434c1c-kube-api-access-klmml\") pod \"84a96c90-0ab3-4482-91b9-cb9282434c1c\" (UID: \"84a96c90-0ab3-4482-91b9-cb9282434c1c\") " Oct 30 06:33:44.027244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.027206 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-model-cache" pod "84a96c90-0ab3-4482-91b9-cb9282434c1c" (UID: "84a96c90-0ab3-4482-91b9-cb9282434c1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:44.027829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.027802 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-tmp-dir" pod "84a96c90-0ab3-4482-91b9-cb9282434c1c" (UID: "84a96c90-0ab3-4482-91b9-cb9282434c1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:44.027951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.027862 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-home" pod "84a96c90-0ab3-4482-91b9-cb9282434c1c" (UID: "84a96c90-0ab3-4482-91b9-cb9282434c1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:44.030163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.030129 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a96c90-0ab3-4482-91b9-cb9282434c1c-kube-api-access-klmml" pod "84a96c90-0ab3-4482-91b9-cb9282434c1c" (UID: "84a96c90-0ab3-4482-91b9-cb9282434c1c"). InnerVolumeSpecName "kube-api-access-klmml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:44.030163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.030152 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a96c90-0ab3-4482-91b9-cb9282434c1c-tls-certs" pod "84a96c90-0ab3-4482-91b9-cb9282434c1c" (UID: "84a96c90-0ab3-4482-91b9-cb9282434c1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:33:44.030687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.030664 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-dshm" pod "84a96c90-0ab3-4482-91b9-cb9282434c1c" (UID: "84a96c90-0ab3-4482-91b9-cb9282434c1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:44.128552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.128295 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:44.128552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.128339 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:44.128552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.128354 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klmml\" (UniqueName: \"kubernetes.io/projected/84a96c90-0ab3-4482-91b9-cb9282434c1c-kube-api-access-klmml\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:44.128552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.128370 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:44.128552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.128386 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a96c90-0ab3-4482-91b9-cb9282434c1c-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:44.128552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.128398 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a96c90-0ab3-4482-91b9-cb9282434c1c-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:33:44.316556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.316517 5001 generic.go:356] "Generic (PLEG): container finished" podID="84a96c90-0ab3-4482-91b9-cb9282434c1c" containerID="8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972" exitCode=0 Oct 30 06:33:44.316763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.316569 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" Oct 30 06:33:44.316763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.316563 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" event={"ID":"84a96c90-0ab3-4482-91b9-cb9282434c1c","Type":"ContainerDied","Data":"8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972"} Oct 30 06:33:44.316763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.316694 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht" event={"ID":"84a96c90-0ab3-4482-91b9-cb9282434c1c","Type":"ContainerDied","Data":"6dc28bea7ec14e1064e796874b192cd8887cd8c65e671730d04b55cfaf310244"} Oct 30 06:33:44.316763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.316723 5001 scope.go:122] "RemoveContainer" containerID="8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972" Oct 30 06:33:44.337756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.337656 5001 scope.go:122] "RemoveContainer" containerID="8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972" Oct 30 06:33:44.338025 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:33:44.338002 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972\": container with ID starting with 8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972 not found: ID does not exist" containerID="8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972" Oct 30 06:33:44.338110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.338039 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972"} err="failed to get container status \"8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972\": rpc error: code = NotFound desc = could not find container \"8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972\": container with ID starting with 8965f69c8a2b102f11b3cb2257a7a3cfcc60a08849c9946f824b0876700f8972 not found: ID does not exist" Oct 30 06:33:44.371516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.371471 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht"] Oct 30 06:33:44.380876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:44.380799 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b9ff93/auth-test-model-b9ff93-kserve-7fb4f48569-p8zht"] Oct 30 06:33:46.132541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:46.132497 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a96c90-0ab3-4482-91b9-cb9282434c1c" path="/var/lib/kubelet/pods/84a96c90-0ab3-4482-91b9-cb9282434c1c/volumes" Oct 30 06:33:52.413651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:52.413609 5001 generic.go:356] "Generic (PLEG): container finished" podID="00363ae6-95cd-4372-9209-487f572af4a9" containerID="7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d" exitCode=0 Oct 30 06:33:52.414118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:52.413687 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" event={"ID":"00363ae6-95cd-4372-9209-487f572af4a9","Type":"ContainerDied","Data":"7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d"} Oct 30 06:33:52.414197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:52.414181 5001 scope.go:122] "RemoveContainer" containerID="7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d" Oct 30 06:33:52.416686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:52.416645 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:52.416787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:52.416754 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:52.416827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:52.416802 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:53.427877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:53.427837 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" event={"ID":"00363ae6-95cd-4372-9209-487f572af4a9","Type":"ContainerStarted","Data":"4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18"} Oct 30 06:33:53.572299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:53.572256 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:33:53.572508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:53.572421 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 06:33:53.572508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:53.572468 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 06:33:53.806853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:53.806741 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:33:54.169656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.168483 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2"] Oct 30 06:33:54.169867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.169703 5001 cpu_manager.go:414] "containerMap: removing container" podUID="84a96c90-0ab3-4482-91b9-cb9282434c1c" containerName="main" Oct 30 06:33:54.169867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.169724 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a96c90-0ab3-4482-91b9-cb9282434c1c" containerName="main" Oct 30 06:33:54.170000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.169896 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a96c90-0ab3-4482-91b9-cb9282434c1c" containerName="main" Oct 30 06:33:54.219704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.216624 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.235847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.223890 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2"] Oct 30 06:33:54.235847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.232278 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2d8df2-openshift-default-dockercfg-nmcfj\"" Oct 30 06:33:54.235847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.232332 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istiod-ca-cert\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.235847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.232374 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-envoy\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.235847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.232498 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-credential-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.235847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.232571 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-certs\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.235847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.232729 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-data\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.235847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.232840 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w62sz\" (UniqueName: \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-kube-api-access-w62sz\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.235847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.232941 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.235847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.232981 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-token\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.235847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.233016 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-podinfo\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.476962 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istiod-ca-cert\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.477029 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-envoy\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.477091 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-credential-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.477164 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-certs\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.477269 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-data\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.477392 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w62sz\" (UniqueName: \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-kube-api-access-w62sz\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.477479 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.477524 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-token\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.477564 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-podinfo\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.508602 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-certs\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.553093 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istiod-ca-cert\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.553391 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-credential-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.564809 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-data\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.565049 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.588623 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.588725 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.605015 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-envoy\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.613784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.612993 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-podinfo\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.620415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.618146 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w62sz\" (UniqueName: \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-kube-api-access-w62sz\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.638509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.637788 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-token\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:54.838620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:54.838587 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:55.317473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.316333 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2"] Oct 30 06:33:55.323699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.323655 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:55.323811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.323749 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:55.323811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.323795 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:55.572881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.572408 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 06:33:55.572881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.572471 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 06:33:55.616744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.616707 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm"] Oct 30 06:33:55.685774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.673064 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.700856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.700818 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm"] Oct 30 06:33:55.727596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.727556 5001 generic.go:356] "Generic (PLEG): container finished" podID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerID="29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a" exitCode=0 Oct 30 06:33:55.727794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.727734 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" event={"ID":"7fdd5e60-c7ac-460d-883a-f4e4961f3735","Type":"ContainerDied","Data":"29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a"} Oct 30 06:33:55.728867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.728844 5001 scope.go:122] "RemoveContainer" containerID="29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a" Oct 30 06:33:55.731183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.731161 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" event={"ID":"8b9fe738-2674-4d41-a255-1c2a057d4ca5","Type":"ContainerStarted","Data":"10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3"} Oct 30 06:33:55.731295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.731203 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" event={"ID":"8b9fe738-2674-4d41-a255-1c2a057d4ca5","Type":"ContainerStarted","Data":"cfb7df7bb8d02acaf2681d95a5990a33be02c89acaf25625d306ef199fefde90"} Oct 30 06:33:55.731939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.731899 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:55.732026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.731996 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:55.732068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.732047 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:55.839965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.839623 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:55.849666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.845122 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" podStartSLOduration=1.845101734 podStartE2EDuration="1.845101734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:55.840702019 +0000 UTC m=+2449.836659030" watchObservedRunningTime="2026-10-30 06:33:55.845101734 +0000 UTC m=+2449.841058747" Oct 30 06:33:55.850510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.850486 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.850730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.850715 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmdvv\" (UniqueName: \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-kube-api-access-kmdvv\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.850850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.850840 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.851054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.851044 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.851161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.851119 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.851370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.851354 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.851500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.851483 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b04d115-9b8e-4fb8-ba65-2512d5065070-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.857710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.854320 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.857889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.857868 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.988335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.986474 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.988335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.987031 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.988335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.987130 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.988335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.987161 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b04d115-9b8e-4fb8-ba65-2512d5065070-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.988335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.987207 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.988335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.987235 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.988335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.987334 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.988335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.987448 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmdvv\" (UniqueName: \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-kube-api-access-kmdvv\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.988335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.987511 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.989072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.988449 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b04d115-9b8e-4fb8-ba65-2512d5065070-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.989072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.986956 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.989072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.988727 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.989072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.988963 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:55.989347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:55.989187 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:56.038510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:56.038091 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:56.043817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:56.041706 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:56.064333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:56.057261 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:56.091401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:56.087742 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmdvv\" (UniqueName: \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-kube-api-access-kmdvv\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:56.339328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:56.335778 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:56.571558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:56.569646 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 06:33:56.571558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:56.569704 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 06:33:56.802676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:56.802625 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" event={"ID":"7fdd5e60-c7ac-460d-883a-f4e4961f3735","Type":"ContainerStarted","Data":"0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2"} Oct 30 06:33:56.890805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:56.890773 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:57.007663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.006873 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm"] Oct 30 06:33:57.045782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.045739 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:57.045864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.045844 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:57.045957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.045894 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:57.231259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.231232 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:33:57.232218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.232191 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:33:57.232273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.232245 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 30 06:33:57.577334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.576737 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 06:33:57.577334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.576804 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 06:33:57.844974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.844886 5001 generic.go:356] "Generic (PLEG): container finished" podID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerID="5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f" exitCode=0 Oct 30 06:33:57.845627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.844956 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" event={"ID":"b2bf08e0-cac0-43ed-847a-3df6d9003e3c","Type":"ContainerDied","Data":"5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f"} Oct 30 06:33:57.846174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.846153 5001 scope.go:122] "RemoveContainer" containerID="5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f" Oct 30 06:33:57.848009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.847983 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" event={"ID":"8b04d115-9b8e-4fb8-ba65-2512d5065070","Type":"ContainerStarted","Data":"9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5"} Oct 30 06:33:57.848182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.848166 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" event={"ID":"8b04d115-9b8e-4fb8-ba65-2512d5065070","Type":"ContainerStarted","Data":"f663f06d135db62a4dac1f67d199d2f245ee3420dc0329d33a42082668693516"} Oct 30 06:33:57.848287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.848260 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:57.848382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.848366 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:57.848448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.848400 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:57.848995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.848980 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:57.850108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.850089 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:33:57.997331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.994971 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:33:57.997331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:57.995540 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:33:58.002515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:58.001412 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" podStartSLOduration=3.001392206 podStartE2EDuration="3.001392206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:57.990190053 +0000 UTC m=+2451.986147067" watchObservedRunningTime="2026-10-30 06:33:58.001392206 +0000 UTC m=+2451.997349219" Oct 30 06:33:58.241342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:58.233768 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:33:58.335792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:58.335634 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 30 06:33:58.350356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:58.336618 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:33:58.350356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:58.343555 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 30 06:33:58.350356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:58.343628 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 30 06:33:58.597395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:58.577791 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 06:33:58.601334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:58.597690 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 06:33:58.971335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:58.967895 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" event={"ID":"b2bf08e0-cac0-43ed-847a-3df6d9003e3c","Type":"ContainerStarted","Data":"d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9"} Oct 30 06:33:59.234391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:59.232821 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:33:59.234391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:59.232886 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 30 06:33:59.338338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:59.336520 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 30 06:33:59.338338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:59.336584 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 30 06:33:59.581387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:59.580479 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body= Oct 30 06:33:59.581387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:59.580547 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" Oct 30 06:33:59.767336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:59.766847 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:33:59.767336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:59.766986 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 30 06:33:59.767336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:33:59.767032 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 30 06:34:00.250394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:00.244762 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:34:00.250394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:00.244832 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 30 06:34:00.339777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:00.339741 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 30 06:34:00.340058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:00.340030 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 30 06:34:00.588735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:00.588641 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:34:00.592508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:00.592483 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:34:00.758192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:00.758155 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 30 06:34:00.758504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:00.758478 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 30 06:34:01.354359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.278580 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:34:01.354359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.278639 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 30 06:34:01.402334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.388526 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 30 06:34:01.402334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.388593 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 30 06:34:01.465755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.465706 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d"] Oct 30 06:34:01.550205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.550161 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:34:01.607398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.605980 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 06:34:01.761335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.729613 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 06:34:01.761335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.746585 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:34:01.762839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.762791 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 06:34:01.764673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.764614 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2d8df2\"" Oct 30 06:34:01.787929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.787120 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2d8df2-dockercfg-jtdl8\"" Oct 30 06:34:01.813467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.813432 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d"] Oct 30 06:34:01.852232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.841818 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 30 06:34:01.852232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.841892 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 30 06:34:01.852494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.852461 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efc4083-009c-45f8-b535-7dff63289742-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d\" (UID: \"8efc4083-009c-45f8-b535-7dff63289742\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:34:01.874295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.852564 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwrl\" (UniqueName: \"kubernetes.io/projected/8efc4083-009c-45f8-b535-7dff63289742-kube-api-access-qcwrl\") pod \"payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d\" (UID: \"8efc4083-009c-45f8-b535-7dff63289742\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:34:02.039330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.038624 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8"] Oct 30 06:34:02.102330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:01.901651 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:34:02.170970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.170882 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8"] Oct 30 06:34:02.171488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.171467 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:34:02.179523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.179491 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efc4083-009c-45f8-b535-7dff63289742-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d\" (UID: \"8efc4083-009c-45f8-b535-7dff63289742\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:34:02.179822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.179791 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwrl\" (UniqueName: \"kubernetes.io/projected/8efc4083-009c-45f8-b535-7dff63289742-kube-api-access-qcwrl\") pod \"payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d\" (UID: \"8efc4083-009c-45f8-b535-7dff63289742\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:34:02.185778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.185645 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efc4083-009c-45f8-b535-7dff63289742-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d\" (UID: \"8efc4083-009c-45f8-b535-7dff63289742\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:34:02.249662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.229177 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwrl\" (UniqueName: \"kubernetes.io/projected/8efc4083-009c-45f8-b535-7dff63289742-kube-api-access-qcwrl\") pod \"payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d\" (UID: \"8efc4083-009c-45f8-b535-7dff63289742\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:34:02.252546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.252514 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:34:02.302100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.302048 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chl8n\" (UniqueName: \"kubernetes.io/projected/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-kube-api-access-chl8n\") pod \"payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8\" (UID: \"75d360de-4c0c-4d3f-bc06-1ea1922e9f43\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:34:02.307896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.307801 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8\" (UID: \"75d360de-4c0c-4d3f-bc06-1ea1922e9f43\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:34:02.312009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.311952 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 30 06:34:02.345476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.345435 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 30 06:34:02.345784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.345755 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 30 06:34:02.424477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.424366 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8\" (UID: \"75d360de-4c0c-4d3f-bc06-1ea1922e9f43\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:34:02.429845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.429815 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chl8n\" (UniqueName: \"kubernetes.io/projected/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-kube-api-access-chl8n\") pod \"payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8\" (UID: \"75d360de-4c0c-4d3f-bc06-1ea1922e9f43\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:34:02.430685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.429554 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8\" (UID: \"75d360de-4c0c-4d3f-bc06-1ea1922e9f43\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:34:02.441533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.441493 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:34:02.523339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.518622 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chl8n\" (UniqueName: \"kubernetes.io/projected/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-kube-api-access-chl8n\") pod \"payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8\" (UID: \"75d360de-4c0c-4d3f-bc06-1ea1922e9f43\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:34:02.523339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.520271 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:34:02.703457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.674211 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:34:02.777335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.776724 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 30 06:34:02.777335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:02.776782 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 30 06:34:03.306788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.305945 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:34:03.306788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.306010 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 30 06:34:03.341910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.341757 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 30 06:34:03.341910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.341843 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 30 06:34:03.605540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.550891 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:34:03.605540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.584630 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d"] Oct 30 06:34:03.631719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.614667 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:34:03.660332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.637507 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 06:34:03.837447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.822007 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 30 06:34:03.837447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.822069 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 30 06:34:03.837447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.822144 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:34:03.837447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:03.822434 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 06:34:04.025202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:04.024985 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" event={"ID":"8efc4083-009c-45f8-b535-7dff63289742","Type":"ContainerStarted","Data":"a5bdae326110c79f1d72c03fa9de5c85db61ae84197858f95e3cd4fb957eec60"} Oct 30 06:34:04.338425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:04.221031 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8"] Oct 30 06:34:04.338425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:04.337845 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 30 06:34:04.338425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:04.337900 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 30 06:34:04.358694 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:34:04.358663 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d360de_4c0c_4d3f_bc06_1ea1922e9f43.slice/crio-bd446deffffb073c0ea81b42f9184f7b394f49d94a0812e70d4b93d2a54c5a9f WatchSource:0}: Error finding container bd446deffffb073c0ea81b42f9184f7b394f49d94a0812e70d4b93d2a54c5a9f: Status 404 returned error can't find the container with id bd446deffffb073c0ea81b42f9184f7b394f49d94a0812e70d4b93d2a54c5a9f Oct 30 06:34:04.459819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:04.459774 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:34:04.681365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:04.674474 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:34:04.695985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:04.682580 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:34:04.899623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:04.828213 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 30 06:34:04.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:04.944842 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 30 06:34:05.200431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.156652 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:34:05.216241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.215833 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:34:05.283501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.283459 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" event={"ID":"75d360de-4c0c-4d3f-bc06-1ea1922e9f43","Type":"ContainerStarted","Data":"2cd7ebec86e493d6bb27627fba5bc8185d9d9b30775015808af77ecb44386432"} Oct 30 06:34:05.284433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.284402 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" event={"ID":"75d360de-4c0c-4d3f-bc06-1ea1922e9f43","Type":"ContainerStarted","Data":"bd446deffffb073c0ea81b42f9184f7b394f49d94a0812e70d4b93d2a54c5a9f"} Oct 30 06:34:05.284651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.284635 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:34:05.289881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.289856 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 06:34:05.305351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.305275 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" event={"ID":"8efc4083-009c-45f8-b535-7dff63289742","Type":"ContainerStarted","Data":"8d2484a8a870f627e85ae4b37fb447e3ee44733afbcded16021dd4e8aa4f108c"} Oct 30 06:34:05.338229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.338194 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 30 06:34:05.338547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.338515 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 30 06:34:05.416436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.416377 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" podStartSLOduration=5.416357581 podStartE2EDuration="5.416357581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:05.401798377 +0000 UTC m=+2459.397755389" watchObservedRunningTime="2026-10-30 06:34:05.416357581 +0000 UTC m=+2459.412314591" Oct 30 06:34:05.502978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.502845 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" podStartSLOduration=5.502824357 podStartE2EDuration="5.502824357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:05.445212619 +0000 UTC m=+2459.441169630" watchObservedRunningTime="2026-10-30 06:34:05.502824357 +0000 UTC m=+2459.498781368" Oct 30 06:34:05.785330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.785217 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 30 06:34:05.786084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:05.786048 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 30 06:34:06.496550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:06.491435 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:34:06.673411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:06.672814 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:34:06.675606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:06.674825 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:34:06.692343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:06.676814 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:34:06.807477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:06.775657 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body= Oct 30 06:34:06.807477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:06.775713 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" Oct 30 06:34:07.628019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.627974 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc"] Oct 30 06:34:07.636337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.635572 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.652720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.652646 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-69f3fa\"/\"agc-cred-8ef0bfb33772\"" Oct 30 06:34:07.653128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.653068 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-69f3fa\"/\"payload-processing-external-model-e2e--e6e7e5a5-dockercfg-ls8fb\"" Oct 30 06:34:07.662479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.660471 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-69f3fa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-69f3fa\"" Oct 30 06:34:07.662479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.660609 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-69f3fa\"/\"kube-root-ca.crt\"" Oct 30 06:34:07.662479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.660732 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-69f3fa\"/\"openshift-service-ca.crt\"" Oct 30 06:34:07.664858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.664829 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc"] Oct 30 06:34:07.671816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.671779 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-69f3fa\"/\"routing-overlay\"" Oct 30 06:34:07.775815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.768141 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.775815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.768364 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.775815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.768568 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srj4\" (UniqueName: \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-kube-api-access-8srj4\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.775815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.768617 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.895381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.891488 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srj4\" (UniqueName: \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-kube-api-access-8srj4\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.895381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.891562 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.895381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.891667 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.895381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.891833 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.895381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.892808 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.895381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.894127 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.938347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.937258 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srj4\" (UniqueName: \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-kube-api-access-8srj4\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.967330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.962508 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:07.971333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:07.967595 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:08.054872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.054096 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:34:08.055126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.055102 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:34:08.062015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.060387 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:34:08.278252 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:34:08.277848 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:34:08.351247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.351218 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc"] Oct 30 06:34:08.352362 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:34:08.352287 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c389a4d_4f6e_4610_bf42_7de433f41ee1.slice/crio-0b90452dd965a28698391f25313c4f953ef89aea378a61e0e7b97d25d7f11936 WatchSource:0}: Error finding container 0b90452dd965a28698391f25313c4f953ef89aea378a61e0e7b97d25d7f11936: Status 404 returned error can't find the container with id 0b90452dd965a28698391f25313c4f953ef89aea378a61e0e7b97d25d7f11936 Oct 30 06:34:08.601419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.601092 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" event={"ID":"9c389a4d-4f6e-4610-bf42-7de433f41ee1","Type":"ContainerStarted","Data":"c3a5eb816ffb42e1cad238c73a25bfef00b74b4010571e8ae652e7e1af372734"} Oct 30 06:34:08.601419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.601152 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" event={"ID":"9c389a4d-4f6e-4610-bf42-7de433f41ee1","Type":"ContainerStarted","Data":"0b90452dd965a28698391f25313c4f953ef89aea378a61e0e7b97d25d7f11936"} Oct 30 06:34:08.623489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.601860 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:08.698205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.698089 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" podStartSLOduration=1.698070484 podStartE2EDuration="1.698070484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:08.690739614 +0000 UTC m=+2462.686696625" watchObservedRunningTime="2026-10-30 06:34:08.698070484 +0000 UTC m=+2462.694027497" Oct 30 06:34:08.699406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.699382 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9"] Oct 30 06:34:08.709176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.709148 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.713526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.713153 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9"] Oct 30 06:34:08.727917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.727890 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"openshift-service-ca.crt\"" Oct 30 06:34:08.728072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.727923 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a0083e\"" Oct 30 06:34:08.728072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.727926 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"agc-cred-f9926dfc3023\"" Oct 30 06:34:08.728072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.727983 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"payload-processing-external-model-e2e--b62900db-dockercfg-v5k9f\"" Oct 30 06:34:08.728072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.728073 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"kube-root-ca.crt\"" Oct 30 06:34:08.728237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.728080 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"routing-overlay\"" Oct 30 06:34:08.728975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.728951 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.729065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.729047 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.729505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.729478 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smqlz\" (UniqueName: \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-kube-api-access-smqlz\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.729693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.729661 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.850333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.849785 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.850333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.849983 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.850333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.850030 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.850333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.850202 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smqlz\" (UniqueName: \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-kube-api-access-smqlz\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.857373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.851399 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.857373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.852031 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.870328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.866884 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:08.924156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:08.924119 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smqlz\" (UniqueName: \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-kube-api-access-smqlz\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:09.025727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:09.025680 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:09.369705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:09.369675 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9"] Oct 30 06:34:09.371055 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:34:09.371023 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ad59f5_6a91_4781_9c89_7802aa3f29c3.slice/crio-bcfe0797d5bc58f3de4618a00042d73e121b52fd16e20070f33a7c2d2ca1f14c WatchSource:0}: Error finding container bcfe0797d5bc58f3de4618a00042d73e121b52fd16e20070f33a7c2d2ca1f14c: Status 404 returned error can't find the container with id bcfe0797d5bc58f3de4618a00042d73e121b52fd16e20070f33a7c2d2ca1f14c Oct 30 06:34:09.610513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:09.610472 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" event={"ID":"d6ad59f5-6a91-4781-9c89-7802aa3f29c3","Type":"ContainerStarted","Data":"4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95"} Oct 30 06:34:09.610704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:09.610539 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" event={"ID":"d6ad59f5-6a91-4781-9c89-7802aa3f29c3","Type":"ContainerStarted","Data":"bcfe0797d5bc58f3de4618a00042d73e121b52fd16e20070f33a7c2d2ca1f14c"} Oct 30 06:34:09.610704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:09.610567 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:09.672065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:09.671921 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" podStartSLOduration=1.67189948 podStartE2EDuration="1.67189948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:09.650598473 +0000 UTC m=+2463.646555485" watchObservedRunningTime="2026-10-30 06:34:09.67189948 +0000 UTC m=+2463.667856491" Oct 30 06:34:11.131934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.131900 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w"] Oct 30 06:34:11.136593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.136571 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:34:11.165720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.165690 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6d1778\"" Oct 30 06:34:11.166001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.165982 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6d1778-dockercfg-bhk94\"" Oct 30 06:34:11.166728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.166702 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w"] Oct 30 06:34:11.217043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.217009 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5j9j\" (UniqueName: \"kubernetes.io/projected/813c0acd-5bc2-42ee-9eac-133d057df75f-kube-api-access-m5j9j\") pod \"payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w\" (UID: \"813c0acd-5bc2-42ee-9eac-133d057df75f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:34:11.217265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.217100 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/813c0acd-5bc2-42ee-9eac-133d057df75f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w\" (UID: \"813c0acd-5bc2-42ee-9eac-133d057df75f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:34:11.323812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.323705 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5j9j\" (UniqueName: \"kubernetes.io/projected/813c0acd-5bc2-42ee-9eac-133d057df75f-kube-api-access-m5j9j\") pod \"payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w\" (UID: \"813c0acd-5bc2-42ee-9eac-133d057df75f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:34:11.325328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.324184 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/813c0acd-5bc2-42ee-9eac-133d057df75f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w\" (UID: \"813c0acd-5bc2-42ee-9eac-133d057df75f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:34:11.339331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.336068 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/813c0acd-5bc2-42ee-9eac-133d057df75f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w\" (UID: \"813c0acd-5bc2-42ee-9eac-133d057df75f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:34:11.368330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.367989 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5j9j\" (UniqueName: \"kubernetes.io/projected/813c0acd-5bc2-42ee-9eac-133d057df75f-kube-api-access-m5j9j\") pod \"payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w\" (UID: \"813c0acd-5bc2-42ee-9eac-133d057df75f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:34:11.454373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.453548 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:34:11.867830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:11.867804 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w"] Oct 30 06:34:11.870262 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:34:11.870233 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813c0acd_5bc2_42ee_9eac_133d057df75f.slice/crio-64fa9f7ac4f5999319a25a3824304126cf03b0f0cd7e7794e9cb86a094954d7a WatchSource:0}: Error finding container 64fa9f7ac4f5999319a25a3824304126cf03b0f0cd7e7794e9cb86a094954d7a: Status 404 returned error can't find the container with id 64fa9f7ac4f5999319a25a3824304126cf03b0f0cd7e7794e9cb86a094954d7a Oct 30 06:34:12.656881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:12.656832 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" event={"ID":"813c0acd-5bc2-42ee-9eac-133d057df75f","Type":"ContainerStarted","Data":"88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388"} Oct 30 06:34:12.657388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:12.656935 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" event={"ID":"813c0acd-5bc2-42ee-9eac-133d057df75f","Type":"ContainerStarted","Data":"64fa9f7ac4f5999319a25a3824304126cf03b0f0cd7e7794e9cb86a094954d7a"} Oct 30 06:34:12.657388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:12.657058 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:34:12.699896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:12.699823 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" podStartSLOduration=1.6998034739999999 podStartE2EDuration="1.699803474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:12.698177227 +0000 UTC m=+2466.694134237" watchObservedRunningTime="2026-10-30 06:34:12.699803474 +0000 UTC m=+2466.695760499" Oct 30 06:34:13.574339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:13.574195 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:34:17.471952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:17.471862 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:34:17.471952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:17.471940 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:34:20.629639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:20.629613 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:34:20.631628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:20.631606 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:34:23.748885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:23.748858 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:34:25.234107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.234054 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c"] Oct 30 06:34:25.238527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.238504 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.283346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.283274 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c"] Oct 30 06:34:25.291240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.291212 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq"] Oct 30 06:34:25.299616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.299587 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.309554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.309517 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq"] Oct 30 06:34:25.351831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.351797 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.352034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.351856 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.352034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.351956 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.352173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.352071 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.352173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.352164 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.352284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.352212 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.352389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.352361 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.352539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.352520 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmkbw\" (UniqueName: \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-kube-api-access-hmkbw\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.352601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.352571 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.453971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.453935 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.454155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454001 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.454155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454069 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.454155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454133 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.454410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454169 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.454410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454294 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.454410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454361 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.454603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454437 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.454603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454470 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.454603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454486 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.454603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454538 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.454603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454580 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.454858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454701 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bc9f11-c202-425e-9382-fa1a6580b6bb-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.454858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454785 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.454858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454832 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.455031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454927 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.455031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.454964 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2ksg\" (UniqueName: \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-kube-api-access-c2ksg\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.455031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.455000 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.455193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.455088 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.455193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.455169 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmkbw\" (UniqueName: \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-kube-api-access-hmkbw\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.455505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.455209 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.455505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.455463 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.455639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.455548 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.457010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.456983 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.457339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.457289 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.476786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.476762 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.478880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.478859 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmkbw\" (UniqueName: \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-kube-api-access-hmkbw\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.556672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.556579 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.556672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.556617 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.556672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.556652 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bc9f11-c202-425e-9382-fa1a6580b6bb-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.556972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.556708 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.556972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.556787 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ksg\" (UniqueName: \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-kube-api-access-c2ksg\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.556972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.556836 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:25.556972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.556925 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.557182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.557002 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.557182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.557054 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.557182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.557121 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.557373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.557051 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.557498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.557469 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.557653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.557588 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.557653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.557625 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.558033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.558007 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bc9f11-c202-425e-9382-fa1a6580b6bb-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.559643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.559613 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.560486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.560335 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.579812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.579784 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ksg\" (UniqueName: \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-kube-api-access-c2ksg\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.580225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.580196 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.631157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.631129 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:25.907354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.904915 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c"] Oct 30 06:34:25.974859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.972618 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:25.974859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.972714 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:25.974859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:25.972759 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:26.032739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:26.032705 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq"] Oct 30 06:34:26.035221 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:34:26.035196 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bc9f11_c202_425e_9382_fa1a6580b6bb.slice/crio-d5c101a3c0c1f12596fe7c1c4ddf1c987d51c341d20627cb52f0949c5d62007e WatchSource:0}: Error finding container d5c101a3c0c1f12596fe7c1c4ddf1c987d51c341d20627cb52f0949c5d62007e: Status 404 returned error can't find the container with id d5c101a3c0c1f12596fe7c1c4ddf1c987d51c341d20627cb52f0949c5d62007e Oct 30 06:34:26.038259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:26.038222 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:26.038366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:26.038294 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:26.038366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:26.038350 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:27.010964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.010913 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" event={"ID":"05bc9f11-c202-425e-9382-fa1a6580b6bb","Type":"ContainerStarted","Data":"03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4"} Oct 30 06:34:27.011539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.010983 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" event={"ID":"05bc9f11-c202-425e-9382-fa1a6580b6bb","Type":"ContainerStarted","Data":"d5c101a3c0c1f12596fe7c1c4ddf1c987d51c341d20627cb52f0949c5d62007e"} Oct 30 06:34:27.015263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.015235 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" event={"ID":"81f33b27-3d5e-4a49-91e1-e44a8a5ce874","Type":"ContainerStarted","Data":"1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c"} Oct 30 06:34:27.015364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.015286 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" event={"ID":"81f33b27-3d5e-4a49-91e1-e44a8a5ce874","Type":"ContainerStarted","Data":"92dc7120d6dace30e463d8da407768723c8544b0927eb397738955e948592338"} Oct 30 06:34:27.073330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.073133 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" podStartSLOduration=2.07310927 podStartE2EDuration="2.07310927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:27.070796427 +0000 UTC m=+2481.066753441" watchObservedRunningTime="2026-10-30 06:34:27.07310927 +0000 UTC m=+2481.069066282" Oct 30 06:34:27.125960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.125670 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" podStartSLOduration=2.125644805 podStartE2EDuration="2.125644805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:27.121793691 +0000 UTC m=+2481.117750702" watchObservedRunningTime="2026-10-30 06:34:27.125644805 +0000 UTC m=+2481.121601814" Oct 30 06:34:27.566330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.561575 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:27.566528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.566422 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 30 06:34:27.566528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.566476 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" podUID="81f33b27-3d5e-4a49-91e1-e44a8a5ce874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 30 06:34:27.631669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.631637 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:27.633088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.633063 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 06:34:27.633227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:27.633109 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" podUID="05bc9f11-c202-425e-9382-fa1a6580b6bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 06:34:28.575333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:28.574793 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 30 06:34:28.575333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:28.574860 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" podUID="81f33b27-3d5e-4a49-91e1-e44a8a5ce874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 30 06:34:28.631779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:28.631743 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 06:34:28.631977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:28.631793 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" podUID="05bc9f11-c202-425e-9382-fa1a6580b6bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 06:34:29.560331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:29.559661 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 30 06:34:29.560331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:29.559721 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" podUID="81f33b27-3d5e-4a49-91e1-e44a8a5ce874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 30 06:34:29.632009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:29.631978 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 06:34:29.632425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:29.632029 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" podUID="05bc9f11-c202-425e-9382-fa1a6580b6bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 06:34:30.573330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:30.560678 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 30 06:34:30.573330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:30.560756 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" podUID="81f33b27-3d5e-4a49-91e1-e44a8a5ce874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 30 06:34:30.631506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:30.631470 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 06:34:30.631685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:30.631528 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" podUID="05bc9f11-c202-425e-9382-fa1a6580b6bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 06:34:31.563330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:31.557842 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body= Oct 30 06:34:31.563330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:31.557904 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" podUID="81f33b27-3d5e-4a49-91e1-e44a8a5ce874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" Oct 30 06:34:31.641329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:31.640691 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 06:34:31.641329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:31.640751 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" podUID="05bc9f11-c202-425e-9382-fa1a6580b6bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 06:34:32.562489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:32.562460 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:32.562882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:32.562854 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:32.563484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:32.563469 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:34:32.637125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:32.637088 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:32.637519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:32.637499 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:32.638189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:32.638170 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:34:38.269006 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:34:38.268968 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:34:40.762081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.762029 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn"] Oct 30 06:34:40.770106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.770080 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.808051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.808018 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn"] Oct 30 06:34:40.848977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.848939 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.849182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.849091 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.849182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.849133 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.849348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.849207 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2n6\" (UniqueName: \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-kube-api-access-8r2n6\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.849348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.849257 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.849348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.849326 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.849535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.849382 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.849535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.849475 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.849655 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.849579 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.951343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.951289 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.951530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.951391 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2n6\" (UniqueName: \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-kube-api-access-8r2n6\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.951530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.951435 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.951530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.951496 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.951708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.951566 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.951708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.951614 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.951708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.951654 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.951897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.951736 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.951964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.951896 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.951964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.951910 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.952088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.951990 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.952207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.952179 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.952456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.952440 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.952613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.952585 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.954625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.954602 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.954786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.954765 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.975980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.975950 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:40.976342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:40.975955 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2n6\" (UniqueName: \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-kube-api-access-8r2n6\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:41.114067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:41.114038 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:41.458950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:41.458917 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn"] Oct 30 06:34:41.460163 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:34:41.460130 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fc25a2_998b_4a99_8ab2_ac5d4b11f415.slice/crio-f3c411b77d3b62e1fb88b374c03f6a263e161d2b8a0602b0cd751ca0fd2678bb WatchSource:0}: Error finding container f3c411b77d3b62e1fb88b374c03f6a263e161d2b8a0602b0cd751ca0fd2678bb: Status 404 returned error can't find the container with id f3c411b77d3b62e1fb88b374c03f6a263e161d2b8a0602b0cd751ca0fd2678bb Oct 30 06:34:41.463257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:41.463225 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:41.463357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:41.463341 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:41.463406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:41.463382 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:42.442332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:42.440977 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" event={"ID":"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415","Type":"ContainerStarted","Data":"dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f"} Oct 30 06:34:42.442332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:42.441096 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" event={"ID":"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415","Type":"ContainerStarted","Data":"f3c411b77d3b62e1fb88b374c03f6a263e161d2b8a0602b0cd751ca0fd2678bb"} Oct 30 06:34:42.533540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:42.533473 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" podStartSLOduration=2.533456256 podStartE2EDuration="2.533456256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:42.516111772 +0000 UTC m=+2496.512068785" watchObservedRunningTime="2026-10-30 06:34:42.533456256 +0000 UTC m=+2496.529413268" Oct 30 06:34:43.116492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:43.115405 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:43.116970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:43.116939 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 06:34:43.117106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:43.116989 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" podUID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 06:34:44.116420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:44.116206 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 06:34:44.116420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:44.116261 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" podUID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 06:34:45.126852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:45.126814 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 06:34:45.127503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:45.127469 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" podUID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 06:34:46.123332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:46.122611 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 06:34:46.123332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:46.122677 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" podUID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 06:34:47.115771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:47.115733 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 06:34:47.116428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:47.116397 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" podUID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 06:34:48.385333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:48.356723 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 06:34:48.386054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:48.385954 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" podUID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 06:34:49.526351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:49.519729 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:49.527443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:49.527407 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:49.528558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:49.528535 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:34:55.831665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.831621 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg"] Oct 30 06:34:55.941843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.941804 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg"] Oct 30 06:34:55.942045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.942015 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:55.997159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.997122 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:55.997364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.997273 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:55.997481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.997456 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:55.997568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.997548 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:55.997805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.997785 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:55.997912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.997897 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:55.998012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.997926 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb9cx\" (UniqueName: \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-kube-api-access-rb9cx\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:55.998059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.998033 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33345dbd-38c7-4372-829a-607ef801ba0b-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:55.998108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:55.998084 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33345dbd-38c7-4372-829a-607ef801ba0b-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.099762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.099667 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.099964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.099766 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.099964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.099791 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb9cx\" (UniqueName: \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-kube-api-access-rb9cx\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.100093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.099954 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33345dbd-38c7-4372-829a-607ef801ba0b-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.100093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.100019 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33345dbd-38c7-4372-829a-607ef801ba0b-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.100198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.100154 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.100257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.100217 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.100365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.100252 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.100432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.100417 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.100494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.100460 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.100642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.100619 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.100880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.100857 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.100969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.100896 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.101195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.101173 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33345dbd-38c7-4372-829a-607ef801ba0b-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.102821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.102790 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.103006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.102987 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33345dbd-38c7-4372-829a-607ef801ba0b-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.123609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.123582 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.125136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.125118 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb9cx\" (UniqueName: \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-kube-api-access-rb9cx\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:56.424243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:56.423986 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:57.045108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:57.045070 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg"] Oct 30 06:34:57.046108 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:34:57.046080 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33345dbd_38c7_4372_829a_607ef801ba0b.slice/crio-8f923fcf0090e898b9dc17fd74bad7696f77deb4cee80051cd0b4ee8ed9e3bf6 WatchSource:0}: Error finding container 8f923fcf0090e898b9dc17fd74bad7696f77deb4cee80051cd0b4ee8ed9e3bf6: Status 404 returned error can't find the container with id 8f923fcf0090e898b9dc17fd74bad7696f77deb4cee80051cd0b4ee8ed9e3bf6 Oct 30 06:34:57.049266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:57.049230 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:57.049367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:57.049300 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:57.049422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:57.049360 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:57.110167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:57.110136 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" event={"ID":"33345dbd-38c7-4372-829a-607ef801ba0b","Type":"ContainerStarted","Data":"8f923fcf0090e898b9dc17fd74bad7696f77deb4cee80051cd0b4ee8ed9e3bf6"} Oct 30 06:34:58.338883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:58.338843 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" event={"ID":"33345dbd-38c7-4372-829a-607ef801ba0b","Type":"ContainerStarted","Data":"579a94f50bcce8ca6cc4dbd77073d4b07a4d1f3712a9055509e0d364a3710660"} Oct 30 06:34:58.406547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:58.406494 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" podStartSLOduration=3.4064774780000002 podStartE2EDuration="3.406477478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:58.396640405 +0000 UTC m=+2512.392597415" watchObservedRunningTime="2026-10-30 06:34:58.406477478 +0000 UTC m=+2512.402434562" Oct 30 06:34:58.424930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:58.424896 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:34:58.426586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:58.426560 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 06:34:58.426719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:58.426609 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 06:34:59.030500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:59.030455 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:34:59.030703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:59.030526 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:34:59.424586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:59.424546 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 06:34:59.425066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:34:59.424616 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 06:35:00.457331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.441714 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 06:35:00.457331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.441797 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 06:35:00.506827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.506790 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr"] Oct 30 06:35:00.554940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.554903 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr"] Oct 30 06:35:00.555373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.555348 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.744555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.744457 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.744555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.744523 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9516fde5-0a00-449c-8872-810fab38705b-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.744818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.744649 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9516fde5-0a00-449c-8872-810fab38705b-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.744818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.744700 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsd6f\" (UniqueName: \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-kube-api-access-rsd6f\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.744818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.744784 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.744818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.744807 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.745052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.744827 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.745052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.744924 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.745052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.744976 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.846845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.846804 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.846857 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.846999 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.847186 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.847272 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.847327 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.847350 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.847419 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.847431 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9516fde5-0a00-449c-8872-810fab38705b-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.847482 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.847521 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9516fde5-0a00-449c-8872-810fab38705b-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.847579 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsd6f\" (UniqueName: \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-kube-api-access-rsd6f\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.847807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.847767 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.848139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.848117 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9516fde5-0a00-449c-8872-810fab38705b-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.850178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.850153 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.850713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.850687 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9516fde5-0a00-449c-8872-810fab38705b-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.879571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.879089 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsd6f\" (UniqueName: \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-kube-api-access-rsd6f\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.884886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.884822 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:00.927140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:00.927104 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:01.425084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:01.425054 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 06:35:01.425221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:01.425173 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 06:35:01.711375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:01.710810 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:01.711375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:01.710906 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:01.711375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:01.710947 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:01.711954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:01.711441 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr"] Oct 30 06:35:02.460270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:02.454743 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 06:35:02.460270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:02.454805 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 06:35:02.521329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:02.519578 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:35:02.561129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:02.558886 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" event={"ID":"9516fde5-0a00-449c-8872-810fab38705b","Type":"ContainerStarted","Data":"9cd1037a59a3e9d06824efd8d59227562bb42ead2ded4614e6a61bb5af8a5410"} Oct 30 06:35:02.561129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:02.558952 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" event={"ID":"9516fde5-0a00-449c-8872-810fab38705b","Type":"ContainerStarted","Data":"6691e95c842663bfa4949af3ef62b09eb0f999df5cfba0be7dd6f0858776e92b"} Oct 30 06:35:02.672635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:02.669296 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" podStartSLOduration=2.669279929 podStartE2EDuration="2.669279929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:02.608856113 +0000 UTC m=+2516.604813123" watchObservedRunningTime="2026-10-30 06:35:02.669279929 +0000 UTC m=+2516.665236940" Oct 30 06:35:02.927671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:02.927633 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:02.929411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:02.929384 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 30 06:35:02.929565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:02.929429 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" podUID="9516fde5-0a00-449c-8872-810fab38705b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 30 06:35:03.425062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:03.425024 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 06:35:03.425279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:03.425080 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 06:35:03.841388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:03.834913 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:35:03.841388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:03.834982 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:35:03.930331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:03.929594 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 30 06:35:03.930331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:03.929663 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" podUID="9516fde5-0a00-449c-8872-810fab38705b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 30 06:35:04.466349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:04.436219 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 06:35:04.466349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:04.436279 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 06:35:04.701332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:04.678428 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:35:04.781391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:04.781264 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 06:35:04.931672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:04.931631 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 30 06:35:04.932340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:04.932275 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" podUID="9516fde5-0a00-449c-8872-810fab38705b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 30 06:35:05.436400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:05.435867 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:35:05.438062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:05.437881 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:35:05.438062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:05.437960 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:35:05.927544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:05.927504 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 30 06:35:05.927777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:05.927612 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" podUID="9516fde5-0a00-449c-8872-810fab38705b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 30 06:35:06.927657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:06.927625 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 30 06:35:06.928328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:06.927681 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" podUID="9516fde5-0a00-449c-8872-810fab38705b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 30 06:35:07.927741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:07.927645 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 30 06:35:07.928385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:07.927741 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" podUID="9516fde5-0a00-449c-8872-810fab38705b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 30 06:35:08.624133 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:35:08.623894 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:35:08.931447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:08.931358 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body= Oct 30 06:35:08.932057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:08.932023 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" podUID="9516fde5-0a00-449c-8872-810fab38705b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" Oct 30 06:35:09.933837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:09.933799 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:09.934365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:09.934339 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:09.935241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:09.935218 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:35:10.140189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.140153 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92"] Oct 30 06:35:10.212017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.211028 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.217141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.216895 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92"] Oct 30 06:35:10.327639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.293973 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.384812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.327983 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.384812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.328225 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dldkq\" (UniqueName: \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-kube-api-access-dldkq\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.384812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.328276 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.384812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.328300 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646d7efa-8189-43f5-803e-7f9706982768-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.384812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.328439 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.384812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.328484 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.384812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.328590 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.384812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.328628 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646d7efa-8189-43f5-803e-7f9706982768-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.430066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.430006 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.430238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.430071 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.430238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.430156 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.430238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.430191 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646d7efa-8189-43f5-803e-7f9706982768-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.430398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.430275 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.430398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.430368 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.430548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.430528 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dldkq\" (UniqueName: \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-kube-api-access-dldkq\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.430641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.430573 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.430641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.430596 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646d7efa-8189-43f5-803e-7f9706982768-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.431461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.431431 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646d7efa-8189-43f5-803e-7f9706982768-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.431784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.431754 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.434084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.432772 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.434084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.433267 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.434350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.434193 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.462679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.461666 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.470979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.470946 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646d7efa-8189-43f5-803e-7f9706982768-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.495428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.495385 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.497512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.497462 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dldkq\" (UniqueName: \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-kube-api-access-dldkq\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:10.728884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:10.728796 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:11.070038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:11.069993 5001 generic.go:356] "Generic (PLEG): container finished" podID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerID="fc0690a4ef141685d5cfde802fa036ac592cedb4dbd57ef0c6085aea73046a5f" exitCode=0 Oct 30 06:35:11.071637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:11.071598 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" event={"ID":"053335c2-28a1-476e-b3d8-9d8bf3c5e005","Type":"ContainerDied","Data":"fc0690a4ef141685d5cfde802fa036ac592cedb4dbd57ef0c6085aea73046a5f"} Oct 30 06:35:11.072061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:11.072038 5001 scope.go:122] "RemoveContainer" containerID="fc0690a4ef141685d5cfde802fa036ac592cedb4dbd57ef0c6085aea73046a5f" Oct 30 06:35:11.074851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:11.074800 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:11.074963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:11.074889 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:11.074963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:11.074936 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:11.276572 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:35:11.276542 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646d7efa_8189_43f5_803e_7f9706982768.slice/crio-62b7fe5ac4d069187cec6ce2844d47ccbac318a93b4cda6e72814c30970874b3 WatchSource:0}: Error finding container 62b7fe5ac4d069187cec6ce2844d47ccbac318a93b4cda6e72814c30970874b3: Status 404 returned error can't find the container with id 62b7fe5ac4d069187cec6ce2844d47ccbac318a93b4cda6e72814c30970874b3 Oct 30 06:35:11.279930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:11.279903 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92"] Oct 30 06:35:11.280815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:11.280777 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:11.280928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:11.280864 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:11.281113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:11.281088 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:12.129833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:12.129791 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" event={"ID":"053335c2-28a1-476e-b3d8-9d8bf3c5e005","Type":"ContainerStarted","Data":"17a315d0616b8b5ee5ede024c521b177c661fcf326eda068238affb106cd41f4"} Oct 30 06:35:12.130347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:12.129863 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" event={"ID":"646d7efa-8189-43f5-803e-7f9706982768","Type":"ContainerStarted","Data":"07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641"} Oct 30 06:35:12.130347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:12.129891 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" event={"ID":"646d7efa-8189-43f5-803e-7f9706982768","Type":"ContainerStarted","Data":"62b7fe5ac4d069187cec6ce2844d47ccbac318a93b4cda6e72814c30970874b3"} Oct 30 06:35:12.195332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:12.194547 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" podStartSLOduration=2.194526327 podStartE2EDuration="2.194526327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:12.177782661 +0000 UTC m=+2526.173739708" watchObservedRunningTime="2026-10-30 06:35:12.194526327 +0000 UTC m=+2526.190483335" Oct 30 06:35:12.356330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:12.353794 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:35:12.356330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:12.353936 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body= Oct 30 06:35:12.356330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:12.353987 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" Oct 30 06:35:12.730069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:12.730010 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:12.731363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:12.731057 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 06:35:12.731363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:12.731102 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" podUID="646d7efa-8189-43f5-803e-7f9706982768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 06:35:13.011592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.011491 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:35:13.011931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.011896 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 06:35:13.135114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.135079 5001 generic.go:356] "Generic (PLEG): container finished" podID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerID="d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c" exitCode=0 Oct 30 06:35:13.135679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.135297 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" event={"ID":"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78","Type":"ContainerDied","Data":"d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c"} Oct 30 06:35:13.136832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.136807 5001 scope.go:122] "RemoveContainer" containerID="d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c" Oct 30 06:35:13.138683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.138651 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:13.138781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.138721 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:13.138781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.138755 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:13.346152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.346121 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body= Oct 30 06:35:13.346390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.346185 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" Oct 30 06:35:13.730497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.730034 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 06:35:13.730497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.730093 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" podUID="646d7efa-8189-43f5-803e-7f9706982768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 06:35:13.890883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.890517 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:35:13.890883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:13.890583 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:35:14.159410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:14.151408 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" event={"ID":"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78","Type":"ContainerStarted","Data":"ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4"} Oct 30 06:35:14.352331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:14.350906 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body= Oct 30 06:35:14.352331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:14.350966 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" Oct 30 06:35:14.747333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:14.744580 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 06:35:14.747333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:14.744641 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" podUID="646d7efa-8189-43f5-803e-7f9706982768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 06:35:15.018396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:15.017873 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:35:15.019340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:15.019057 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:35:15.019340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:15.019111 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 06:35:15.386334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:15.381474 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body= Oct 30 06:35:15.386334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:15.381543 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" Oct 30 06:35:15.730371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:15.729793 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 06:35:15.730371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:15.729872 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" podUID="646d7efa-8189-43f5-803e-7f9706982768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 06:35:16.013448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:16.008805 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:35:16.013448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:16.008873 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 06:35:16.351845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:16.351754 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body= Oct 30 06:35:16.352139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:16.352106 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" Oct 30 06:35:16.739334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:16.738417 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 06:35:16.739334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:16.738482 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" podUID="646d7efa-8189-43f5-803e-7f9706982768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 06:35:17.008346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:17.008224 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:35:17.015439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:17.015390 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 06:35:17.346285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:17.346201 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body= Oct 30 06:35:17.346285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:17.346255 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" Oct 30 06:35:17.742163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:17.742064 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body= Oct 30 06:35:17.742163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:17.742125 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" podUID="646d7efa-8189-43f5-803e-7f9706982768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" Oct 30 06:35:17.818251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:17.804693 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:35:18.007842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:18.007742 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:35:18.008546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:18.008114 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 06:35:18.466947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:18.439371 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:35:18.530743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:18.530391 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:35:18.735097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:18.735017 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:18.735619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:18.735563 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:18.736292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:18.736273 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:35:19.011499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:19.009776 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:35:19.011499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:19.009851 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 06:35:20.007856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:20.007818 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:35:20.008150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:20.007885 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Oct 30 06:35:21.012682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:21.012647 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:35:21.014188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:21.013191 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:35:21.014188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:21.013986 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:35:21.557637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:21.557606 5001 generic.go:356] "Generic (PLEG): container finished" podID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerID="10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3" exitCode=0 Oct 30 06:35:21.557821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:21.557684 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" event={"ID":"8b9fe738-2674-4d41-a255-1c2a057d4ca5","Type":"ContainerDied","Data":"10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3"} Oct 30 06:35:21.558476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:21.558454 5001 scope.go:122] "RemoveContainer" containerID="10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3" Oct 30 06:35:21.560492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:21.560458 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:21.560610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:21.560533 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:21.560610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:21.560568 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:22.596393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:22.596357 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" event={"ID":"8b9fe738-2674-4d41-a255-1c2a057d4ca5","Type":"ContainerStarted","Data":"583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78"} Oct 30 06:35:22.839556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:22.839521 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:35:22.840230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:22.840207 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 06:35:22.840286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:22.840268 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 06:35:23.839719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:23.839688 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 06:35:23.840246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:23.840212 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 06:35:24.839287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:24.839251 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 06:35:24.839493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:24.839323 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 06:35:25.839799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:25.839756 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 06:35:25.840370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:25.839819 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 06:35:26.839324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:26.839283 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 06:35:26.839525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:26.839371 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 06:35:27.842721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:27.842684 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" start-of-body= Oct 30 06:35:27.843345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:27.843287 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.61:15021/healthz/ready\": dial tcp 10.132.1.61:15021: connect: connection refused" Oct 30 06:35:27.853453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:27.853418 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:35:29.441334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:29.435678 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:35:29.453839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:29.453806 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:35:30.561034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.560993 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775"] Oct 30 06:35:30.569879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.569851 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:30.572033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.571908 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775"] Oct 30 06:35:30.601329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.596335 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:35:30.601329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.596443 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:35:30.601329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.596587 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:35:30.678381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.661048 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-a88d7d-zf775\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:30.678381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.661577 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl285\" (UniqueName: \"kubernetes.io/projected/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-kube-api-access-fl285\") pod \"maas-api-revoke-keys-e2e-autores-a88d7d-zf775\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:30.678381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.661647 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-a88d7d-zf775\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:30.700607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.700570 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:35:30.763298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.763255 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl285\" (UniqueName: \"kubernetes.io/projected/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-kube-api-access-fl285\") pod \"maas-api-revoke-keys-e2e-autores-a88d7d-zf775\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:30.763608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.763579 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-a88d7d-zf775\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:30.763741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.763725 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-a88d7d-zf775\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:30.764546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.764411 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-a88d7d-zf775\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:30.766568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.766540 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-a88d7d-zf775\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:30.787103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.787073 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl285\" (UniqueName: \"kubernetes.io/projected/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-kube-api-access-fl285\") pod \"maas-api-revoke-keys-e2e-autores-a88d7d-zf775\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:30.889380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:30.889343 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:31.214060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:31.213979 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775"] Oct 30 06:35:31.220590 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:35:31.220562 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6caf70_f9fa_4ab8_9a3a_a18950f96a5b.slice/crio-a48c1d2bf7f6d448b9e6dc7702044adc29050cd8df563fddf664b94af77a665c WatchSource:0}: Error finding container a48c1d2bf7f6d448b9e6dc7702044adc29050cd8df563fddf664b94af77a665c: Status 404 returned error can't find the container with id a48c1d2bf7f6d448b9e6dc7702044adc29050cd8df563fddf664b94af77a665c Oct 30 06:35:31.851841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:31.851731 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" event={"ID":"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b","Type":"ContainerStarted","Data":"1b3f4bc64e013a698e90a35c7f59a84e4019ce53d4fee8b28f227596cf19be8d"} Oct 30 06:35:31.851841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:31.851799 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" event={"ID":"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b","Type":"ContainerStarted","Data":"a48c1d2bf7f6d448b9e6dc7702044adc29050cd8df563fddf664b94af77a665c"} Oct 30 06:35:31.892746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:31.892687 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" podStartSLOduration=1.892671615 podStartE2EDuration="1.892671615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:31.890266624 +0000 UTC m=+2545.886223633" watchObservedRunningTime="2026-10-30 06:35:31.892671615 +0000 UTC m=+2545.888628627" Oct 30 06:35:32.866877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:32.866841 5001 generic.go:356] "Generic (PLEG): container finished" podID="3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b" containerID="1b3f4bc64e013a698e90a35c7f59a84e4019ce53d4fee8b28f227596cf19be8d" exitCode=0 Oct 30 06:35:32.867387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:32.866993 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" event={"ID":"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b","Type":"ContainerDied","Data":"1b3f4bc64e013a698e90a35c7f59a84e4019ce53d4fee8b28f227596cf19be8d"} Oct 30 06:35:34.088820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.088771 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:34.141510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.141469 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-openshift-ca\") pod \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " Oct 30 06:35:34.141842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.141819 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-xks-ca\") pod \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " Oct 30 06:35:34.141985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.141881 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-kube-api-access-fl285\" (UniqueName: \"kubernetes.io/projected/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-kube-api-access-fl285\") pod \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\" (UID: \"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b\") " Oct 30 06:35:34.141985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.141918 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-openshift-ca" pod "3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b" (UID: "3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:35:34.142596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.142572 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:34.145351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.145325 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-kube-api-access-fl285" pod "3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b" (UID: "3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b"). InnerVolumeSpecName "kube-api-access-fl285". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:34.145670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.145640 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-xks-ca" pod "3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b" (UID: "3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:35:34.244185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.244150 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:34.244185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.244180 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl285\" (UniqueName: \"kubernetes.io/projected/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b-kube-api-access-fl285\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:34.966360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.955360 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9"] Oct 30 06:35:34.968266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.968226 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b" containerName="revoke-keys" Oct 30 06:35:34.968491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.968471 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b" containerName="revoke-keys" Oct 30 06:35:34.968872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.968850 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b" containerName="revoke-keys" Oct 30 06:35:34.981781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.981748 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" Oct 30 06:35:34.997359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.997270 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775" event={"ID":"3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b","Type":"ContainerDied","Data":"a48c1d2bf7f6d448b9e6dc7702044adc29050cd8df563fddf664b94af77a665c"} Oct 30 06:35:34.997568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.997393 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a48c1d2bf7f6d448b9e6dc7702044adc29050cd8df563fddf664b94af77a665c" Oct 30 06:35:34.997568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:34.997496 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.054382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.009982 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9"] Oct 30 06:35:35.054382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.020782 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d8df2\"" Oct 30 06:35:35.054382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.020911 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"routing-overlay\"" Oct 30 06:35:35.054382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.020922 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"openshift-service-ca.crt\"" Oct 30 06:35:35.054382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.021018 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"agc-cred-c41c628c8b52\"" Oct 30 06:35:35.054382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.021143 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"payload-processing-external-model-e2e--17d17c8c-dockercfg-7b6bq\"" Oct 30 06:35:35.092342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.077200 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"kube-root-ca.crt\"" Oct 30 06:35:35.092342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.082020 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.092342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.082115 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.092342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.082205 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t65gg\" (UniqueName: \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-kube-api-access-t65gg\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.092342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.082356 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.184475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.184425 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t65gg\" (UniqueName: \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-kube-api-access-t65gg\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.184698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.184602 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.184865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.184834 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.184963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.184941 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.186916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.186882 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.187097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.187073 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.228963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.228878 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.286102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.286066 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t65gg\" (UniqueName: \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-kube-api-access-t65gg\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.435774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.435738 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:35.926162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.926131 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9"] Oct 30 06:35:35.927361 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:35:35.927287 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03fe03b_18ae_4f95_b729_09f0a9af4f58.slice/crio-a56736d9195c64064aad1a196df44cf7312bd4441dcee12d06e13f159b416ff7 WatchSource:0}: Error finding container a56736d9195c64064aad1a196df44cf7312bd4441dcee12d06e13f159b416ff7: Status 404 returned error can't find the container with id a56736d9195c64064aad1a196df44cf7312bd4441dcee12d06e13f159b416ff7 Oct 30 06:35:35.991981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:35.991943 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" event={"ID":"c03fe03b-18ae-4f95-b729-09f0a9af4f58","Type":"ContainerStarted","Data":"a56736d9195c64064aad1a196df44cf7312bd4441dcee12d06e13f159b416ff7"} Oct 30 06:35:37.004155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:37.004119 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" event={"ID":"c03fe03b-18ae-4f95-b729-09f0a9af4f58","Type":"ContainerStarted","Data":"88ea7b36e11f03994164c0baee1c47f2044249e06923a707665914c77488e50d"} Oct 30 06:35:37.004729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:37.004214 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:37.051136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:37.051070 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" podStartSLOduration=3.051049989 podStartE2EDuration="3.051049989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:37.044588039 +0000 UTC m=+2551.040545048" watchObservedRunningTime="2026-10-30 06:35:37.051049989 +0000 UTC m=+2551.047007000" Oct 30 06:35:38.272974 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:35:38.272917 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:35:41.847334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:41.844095 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8"] Oct 30 06:35:41.863430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:41.848229 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" containerID="cri-o://c65c857a335be8d62275e352a1289212bae86a62fc74d80783b756180ec652fa" gracePeriod=30 Oct 30 06:35:47.204809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.204776 5001 generic.go:356] "Generic (PLEG): container finished" podID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerID="c65c857a335be8d62275e352a1289212bae86a62fc74d80783b756180ec652fa" exitCode=0 Oct 30 06:35:47.205218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.204844 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" event={"ID":"b9396bf9-d598-4311-84fa-28f84fa4c341","Type":"ContainerDied","Data":"c65c857a335be8d62275e352a1289212bae86a62fc74d80783b756180ec652fa"} Oct 30 06:35:47.336035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.336009 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:35:47.364747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.364711 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-kube-api-access-x7kxx\" (UniqueName: \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-kube-api-access-x7kxx\") pod \"b9396bf9-d598-4311-84fa-28f84fa4c341\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " Oct 30 06:35:47.364931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.364882 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-certs\") pod \"b9396bf9-d598-4311-84fa-28f84fa4c341\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " Oct 30 06:35:47.364931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.364923 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-data\") pod \"b9396bf9-d598-4311-84fa-28f84fa4c341\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " Oct 30 06:35:47.365018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.364950 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-token\" (UniqueName: \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-token\") pod \"b9396bf9-d598-4311-84fa-28f84fa4c341\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " Oct 30 06:35:47.365163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.365146 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-credential-socket\") pod \"b9396bf9-d598-4311-84fa-28f84fa4c341\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " Oct 30 06:35:47.365221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.365212 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-podinfo\") pod \"b9396bf9-d598-4311-84fa-28f84fa4c341\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " Oct 30 06:35:47.365276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.365215 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-data" pod "b9396bf9-d598-4311-84fa-28f84fa4c341" (UID: "b9396bf9-d598-4311-84fa-28f84fa4c341"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:47.365276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.365235 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-certs" pod "b9396bf9-d598-4311-84fa-28f84fa4c341" (UID: "b9396bf9-d598-4311-84fa-28f84fa4c341"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:47.365561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.365273 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9396bf9-d598-4311-84fa-28f84fa4c341-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9396bf9-d598-4311-84fa-28f84fa4c341-istiod-ca-cert\") pod \"b9396bf9-d598-4311-84fa-28f84fa4c341\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " Oct 30 06:35:47.365561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.365335 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-socket\") pod \"b9396bf9-d598-4311-84fa-28f84fa4c341\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " Oct 30 06:35:47.365561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.365428 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-envoy\") pod \"b9396bf9-d598-4311-84fa-28f84fa4c341\" (UID: \"b9396bf9-d598-4311-84fa-28f84fa4c341\") " Oct 30 06:35:47.365561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.365448 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-credential-socket" pod "b9396bf9-d598-4311-84fa-28f84fa4c341" (UID: "b9396bf9-d598-4311-84fa-28f84fa4c341"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:47.365814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.365732 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9396bf9-d598-4311-84fa-28f84fa4c341-istiod-ca-cert" pod "b9396bf9-d598-4311-84fa-28f84fa4c341" (UID: "b9396bf9-d598-4311-84fa-28f84fa4c341"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:35:47.366225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.366204 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:47.366559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.366543 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:47.366681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.366669 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:47.366791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.366778 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9396bf9-d598-4311-84fa-28f84fa4c341-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:47.366882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.366423 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-socket" pod "b9396bf9-d598-4311-84fa-28f84fa4c341" (UID: "b9396bf9-d598-4311-84fa-28f84fa4c341"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:47.367771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.367744 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-kube-api-access-x7kxx" pod "b9396bf9-d598-4311-84fa-28f84fa4c341" (UID: "b9396bf9-d598-4311-84fa-28f84fa4c341"). InnerVolumeSpecName "kube-api-access-x7kxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:47.368211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.368185 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-envoy" pod "b9396bf9-d598-4311-84fa-28f84fa4c341" (UID: "b9396bf9-d598-4311-84fa-28f84fa4c341"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:47.368453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.368432 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-podinfo" pod "b9396bf9-d598-4311-84fa-28f84fa4c341" (UID: "b9396bf9-d598-4311-84fa-28f84fa4c341"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:35:47.368550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.368455 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-token" pod "b9396bf9-d598-4311-84fa-28f84fa4c341" (UID: "b9396bf9-d598-4311-84fa-28f84fa4c341"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:47.468678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.468630 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:47.468678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.468670 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:47.468678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.468683 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:47.469211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.468697 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7kxx\" (UniqueName: \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-kube-api-access-x7kxx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:47.469211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:47.468711 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9396bf9-d598-4311-84fa-28f84fa4c341-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:48.016785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:48.016758 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:35:48.290955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:48.286159 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" event={"ID":"b9396bf9-d598-4311-84fa-28f84fa4c341","Type":"ContainerDied","Data":"5c6cfa646b6bb9b826c7460b80236ea24d986cb57987fcb585e7918b5fd2bda8"} Oct 30 06:35:48.290955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:48.286241 5001 scope.go:122] "RemoveContainer" containerID="c65c857a335be8d62275e352a1289212bae86a62fc74d80783b756180ec652fa" Oct 30 06:35:48.290955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:48.286444 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8" Oct 30 06:35:48.307539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:48.307513 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:35:48.345987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:48.345943 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8"] Oct 30 06:35:48.369426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:48.369385 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-4d2p8"] Oct 30 06:35:50.120802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:50.120768 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" path="/var/lib/kubelet/pods/b9396bf9-d598-4311-84fa-28f84fa4c341/volumes" Oct 30 06:35:53.034227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:53.034189 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9"] Oct 30 06:35:53.045431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:53.045401 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ecfb0f-kx9f9"] Oct 30 06:35:53.583800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:53.583761 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z"] Oct 30 06:35:53.584079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:53.584050 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" containerID="cri-o://3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6" gracePeriod=30 Oct 30 06:35:54.119946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:54.119910 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f321b71b-f01f-40fd-b934-05bece520a17" path="/var/lib/kubelet/pods/f321b71b-f01f-40fd-b934-05bece520a17/volumes" Oct 30 06:35:55.272616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.272581 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv"] Oct 30 06:35:55.274057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.274031 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" Oct 30 06:35:55.274190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.274060 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" Oct 30 06:35:55.274336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.274322 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9396bf9-d598-4311-84fa-28f84fa4c341" containerName="istio-proxy" Oct 30 06:35:55.283102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.283076 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.312530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.312489 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv"] Oct 30 06:35:55.369019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.368981 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.369253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.369047 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.369253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.369119 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf8f5\" (UniqueName: \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-kube-api-access-nf8f5\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.369253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.369199 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.369526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.369347 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.369526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.369398 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.369526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.369451 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.369526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.369495 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.369526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.369521 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.483339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.476760 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.483339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.476836 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.483339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.476915 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf8f5\" (UniqueName: \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-kube-api-access-nf8f5\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.483339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.476999 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.483339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.477118 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.483339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.477176 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.483339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.477234 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.483339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.477283 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.483339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.477333 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.495858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.484871 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.495858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.491587 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.495858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.491653 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.495858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.492024 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.502661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.502576 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.524039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.523678 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.524513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.524443 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.524653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.524628 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.526583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.526557 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf8f5\" (UniqueName: \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-kube-api-access-nf8f5\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.614089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.614053 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:55.966190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.966149 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv"] Oct 30 06:35:55.968063 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:35:55.968032 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2602925_a5f8_4e4c_b6b7_ea06c8068aff.slice/crio-f45440975334c3d7602a3bd4de37a69ae5be8d2eb5f219bfde42773357f09f83 WatchSource:0}: Error finding container f45440975334c3d7602a3bd4de37a69ae5be8d2eb5f219bfde42773357f09f83: Status 404 returned error can't find the container with id f45440975334c3d7602a3bd4de37a69ae5be8d2eb5f219bfde42773357f09f83 Oct 30 06:35:55.973484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.973418 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:55.973843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.973811 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:55.974125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:55.974101 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:56.397349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:56.397256 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" event={"ID":"a2602925-a5f8-4e4c-b6b7-ea06c8068aff","Type":"ContainerStarted","Data":"a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9"} Oct 30 06:35:56.397349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:56.397342 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" event={"ID":"a2602925-a5f8-4e4c-b6b7-ea06c8068aff","Type":"ContainerStarted","Data":"f45440975334c3d7602a3bd4de37a69ae5be8d2eb5f219bfde42773357f09f83"} Oct 30 06:35:56.444824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:56.444754 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" podStartSLOduration=1.444740251 podStartE2EDuration="1.444740251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:56.441069879 +0000 UTC m=+2570.437026888" watchObservedRunningTime="2026-10-30 06:35:56.444740251 +0000 UTC m=+2570.440697336" Oct 30 06:35:56.615077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:56.615040 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:35:57.616648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:57.616613 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 06:35:57.617100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:57.616668 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" podUID="a2602925-a5f8-4e4c-b6b7-ea06c8068aff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 06:35:58.616605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:58.616561 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 06:35:58.616751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:58.616626 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" podUID="a2602925-a5f8-4e4c-b6b7-ea06c8068aff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 06:35:58.998992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:58.998961 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:35:59.101642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.100365 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-data\") pod \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " Oct 30 06:35:59.101642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.100429 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-socket\") pod \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " Oct 30 06:35:59.101642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.100494 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-kube-api-access-krft7\" (UniqueName: \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-kube-api-access-krft7\") pod \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " Oct 30 06:35:59.101642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.100567 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istiod-ca-cert\") pod \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " Oct 30 06:35:59.101642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.100624 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-podinfo\") pod \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " Oct 30 06:35:59.101642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.100669 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-certs\") pod \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " Oct 30 06:35:59.101642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.100787 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-token\" (UniqueName: \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-token\") pod \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " Oct 30 06:35:59.101642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.100846 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-envoy\") pod \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " Oct 30 06:35:59.101642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.100905 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-credential-socket\") pod \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\" (UID: \"7be6aa98-42b5-46d3-95bc-a69ee605b47e\") " Oct 30 06:35:59.102283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.101862 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-credential-socket" pod "7be6aa98-42b5-46d3-95bc-a69ee605b47e" (UID: "7be6aa98-42b5-46d3-95bc-a69ee605b47e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.102418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.102392 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-socket" pod "7be6aa98-42b5-46d3-95bc-a69ee605b47e" (UID: "7be6aa98-42b5-46d3-95bc-a69ee605b47e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.102931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.102897 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-data" pod "7be6aa98-42b5-46d3-95bc-a69ee605b47e" (UID: "7be6aa98-42b5-46d3-95bc-a69ee605b47e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.103237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.103215 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istiod-ca-cert" pod "7be6aa98-42b5-46d3-95bc-a69ee605b47e" (UID: "7be6aa98-42b5-46d3-95bc-a69ee605b47e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:35:59.103456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.103432 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-certs" pod "7be6aa98-42b5-46d3-95bc-a69ee605b47e" (UID: "7be6aa98-42b5-46d3-95bc-a69ee605b47e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.106032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.105998 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-podinfo" pod "7be6aa98-42b5-46d3-95bc-a69ee605b47e" (UID: "7be6aa98-42b5-46d3-95bc-a69ee605b47e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:35:59.107929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.107907 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-kube-api-access-krft7" pod "7be6aa98-42b5-46d3-95bc-a69ee605b47e" (UID: "7be6aa98-42b5-46d3-95bc-a69ee605b47e"). InnerVolumeSpecName "kube-api-access-krft7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:59.114244 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.114144 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-token" pod "7be6aa98-42b5-46d3-95bc-a69ee605b47e" (UID: "7be6aa98-42b5-46d3-95bc-a69ee605b47e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:59.114644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.114494 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-envoy" pod "7be6aa98-42b5-46d3-95bc-a69ee605b47e" (UID: "7be6aa98-42b5-46d3-95bc-a69ee605b47e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.203445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.203343 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.203643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.203485 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.203643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.203512 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.203643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.203522 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.203643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.203531 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.203643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.203540 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krft7\" (UniqueName: \"kubernetes.io/projected/7be6aa98-42b5-46d3-95bc-a69ee605b47e-kube-api-access-krft7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.203643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.203549 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.203643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.203560 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be6aa98-42b5-46d3-95bc-a69ee605b47e-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.203643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.203568 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be6aa98-42b5-46d3-95bc-a69ee605b47e-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.450986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.450955 5001 generic.go:356] "Generic (PLEG): container finished" podID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerID="3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6" exitCode=0 Oct 30 06:35:59.451143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.451065 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" Oct 30 06:35:59.451143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.451092 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" event={"ID":"7be6aa98-42b5-46d3-95bc-a69ee605b47e","Type":"ContainerDied","Data":"3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6"} Oct 30 06:35:59.451260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.451151 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z" event={"ID":"7be6aa98-42b5-46d3-95bc-a69ee605b47e","Type":"ContainerDied","Data":"5154a0fad2b91c4454135957feec003bdbd206c787cc9f0715f73ebe066753df"} Oct 30 06:35:59.451260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.451182 5001 scope.go:122] "RemoveContainer" containerID="3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6" Oct 30 06:35:59.509127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.509097 5001 scope.go:122] "RemoveContainer" containerID="3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6" Oct 30 06:35:59.509609 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:35:59.509577 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6\": container with ID starting with 3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6 not found: ID does not exist" containerID="3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6" Oct 30 06:35:59.509737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.509618 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6"} err="failed to get container status \"3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6\": rpc error: code = NotFound desc = could not find container \"3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6\": container with ID starting with 3a24f052f038f22bd60dfd79854357d6a53d69fa720ee3c0134b2c73313a56e6 not found: ID does not exist" Oct 30 06:35:59.546007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.545976 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z"] Oct 30 06:35:59.551262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.551231 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-67q6z"] Oct 30 06:35:59.614959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.614929 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 06:35:59.615132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:35:59.614981 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" podUID="a2602925-a5f8-4e4c-b6b7-ea06c8068aff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 06:36:00.121422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:00.121388 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" path="/var/lib/kubelet/pods/7be6aa98-42b5-46d3-95bc-a69ee605b47e/volumes" Oct 30 06:36:00.615009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:00.614974 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 06:36:00.615197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:00.615035 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" podUID="a2602925-a5f8-4e4c-b6b7-ea06c8068aff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 06:36:01.614868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:01.614831 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Oct 30 06:36:01.615321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:01.614896 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" podUID="a2602925-a5f8-4e4c-b6b7-ea06c8068aff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Oct 30 06:36:02.620148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:02.620112 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:36:02.620700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:02.620615 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:36:02.621521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:02.621499 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:36:03.050578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:03.050481 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl"] Oct 30 06:36:03.058098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:03.057993 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d8da6f-xvwfl"] Oct 30 06:36:04.135800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:04.135757 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884a0db4-39db-4613-a5fb-6fc464035d0e" path="/var/lib/kubelet/pods/884a0db4-39db-4613-a5fb-6fc464035d0e/volumes" Oct 30 06:36:08.269702 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:36:08.269656 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:36:10.125967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.125929 5001 scope.go:122] "RemoveContainer" containerID="bc871d1d032940840b4a81d1c0cc19d4aca47da0364065d25dd65a832642eca4" Oct 30 06:36:10.317714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.317681 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5"] Oct 30 06:36:10.318647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.318630 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" Oct 30 06:36:10.318708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.318649 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" Oct 30 06:36:10.318785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.318777 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be6aa98-42b5-46d3-95bc-a69ee605b47e" containerName="istio-proxy" Oct 30 06:36:10.325258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.325220 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.357359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.356384 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5"] Oct 30 06:36:10.493406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.493318 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80da823-bfbb-4b6e-9d73-b959cd28288f-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.493406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.493387 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.493631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.493523 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.493631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.493572 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.493631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.493613 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.493770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.493729 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.493935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.493918 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.493989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.493974 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx2sj\" (UniqueName: \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-kube-api-access-kx2sj\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.494103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.494080 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596031 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596135 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596200 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596254 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596474 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596505 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596524 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596534 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2sj\" (UniqueName: \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-kube-api-access-kx2sj\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596633 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596641 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596707 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80da823-bfbb-4b6e-9d73-b959cd28288f-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596784 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.596939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.596805 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.597299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.597273 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80da823-bfbb-4b6e-9d73-b959cd28288f-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.599846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.599818 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.600132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.600111 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.619683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.619493 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.619837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.619766 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2sj\" (UniqueName: \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-kube-api-access-kx2sj\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:10.664682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:10.664646 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:11.002688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:11.002654 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5"] Oct 30 06:36:11.003734 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:36:11.003707 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80da823_bfbb_4b6e_9d73_b959cd28288f.slice/crio-653e9437431446032508adc92b9c41cff6cd706d1847e7655d31cc129c2f930b WatchSource:0}: Error finding container 653e9437431446032508adc92b9c41cff6cd706d1847e7655d31cc129c2f930b: Status 404 returned error can't find the container with id 653e9437431446032508adc92b9c41cff6cd706d1847e7655d31cc129c2f930b Oct 30 06:36:11.006668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:11.006635 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:11.006746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:11.006704 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:11.006746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:11.006735 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:11.605874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:11.605830 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" event={"ID":"c80da823-bfbb-4b6e-9d73-b959cd28288f","Type":"ContainerStarted","Data":"eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2"} Oct 30 06:36:11.606417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:11.605921 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" event={"ID":"c80da823-bfbb-4b6e-9d73-b959cd28288f","Type":"ContainerStarted","Data":"653e9437431446032508adc92b9c41cff6cd706d1847e7655d31cc129c2f930b"} Oct 30 06:36:11.663518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:11.663454 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" podStartSLOduration=1.663436619 podStartE2EDuration="1.663436619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:11.661398013 +0000 UTC m=+2585.657355025" watchObservedRunningTime="2026-10-30 06:36:11.663436619 +0000 UTC m=+2585.659393629" Oct 30 06:36:11.666319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:11.666274 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:12.677333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:12.673168 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body= Oct 30 06:36:12.677333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:12.673245 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" podUID="c80da823-bfbb-4b6e-9d73-b959cd28288f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" Oct 30 06:36:13.675735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:13.675697 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body= Oct 30 06:36:13.697559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:13.697517 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" podUID="c80da823-bfbb-4b6e-9d73-b959cd28288f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" Oct 30 06:36:14.671330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:14.670409 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body= Oct 30 06:36:14.671330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:14.670472 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" podUID="c80da823-bfbb-4b6e-9d73-b959cd28288f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" Oct 30 06:36:15.673332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:15.672736 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body= Oct 30 06:36:15.673332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:15.672798 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" podUID="c80da823-bfbb-4b6e-9d73-b959cd28288f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" Oct 30 06:36:16.683334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:16.681467 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body= Oct 30 06:36:16.683334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:16.681561 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" podUID="c80da823-bfbb-4b6e-9d73-b959cd28288f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" Oct 30 06:36:17.668466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:17.665613 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" start-of-body= Oct 30 06:36:17.668466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:17.665675 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" podUID="c80da823-bfbb-4b6e-9d73-b959cd28288f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.77:15021/healthz/ready\": dial tcp 10.132.1.77:15021: connect: connection refused" Oct 30 06:36:18.671413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:18.671384 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:18.671928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:18.671733 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:18.672544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:18.672519 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:36:23.754935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:23.754883 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775"] Oct 30 06:36:23.775397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:23.775360 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a88d7d-zf775"] Oct 30 06:36:24.120017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.119981 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b" path="/var/lib/kubelet/pods/3c6caf70-f9fa-4ab8-9a3a-a18950f96a5b/volumes" Oct 30 06:36:24.502460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.502373 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l"] Oct 30 06:36:24.508751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.508728 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:24.527216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.527185 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l"] Oct 30 06:36:24.532196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.532171 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:36:24.532196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.532171 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:36:24.576464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.576426 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:36:24.576464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.576426 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:36:24.669737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.669706 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xd6\" (UniqueName: \"kubernetes.io/projected/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-kube-api-access-h6xd6\") pod \"maas-api-revoke-sub-ceb4fd4f-hzq2l\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:24.669965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.669867 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceb4fd4f-hzq2l\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:24.670045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.670016 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceb4fd4f-hzq2l\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:24.772027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.771935 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xd6\" (UniqueName: \"kubernetes.io/projected/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-kube-api-access-h6xd6\") pod \"maas-api-revoke-sub-ceb4fd4f-hzq2l\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:24.772535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.772077 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceb4fd4f-hzq2l\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:24.772535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.772279 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceb4fd4f-hzq2l\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:24.772903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.772879 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceb4fd4f-hzq2l\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:24.774707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.774680 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceb4fd4f-hzq2l\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:24.795081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.795044 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xd6\" (UniqueName: \"kubernetes.io/projected/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-kube-api-access-h6xd6\") pod \"maas-api-revoke-sub-ceb4fd4f-hzq2l\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:24.825767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:24.825742 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:25.249635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:25.249602 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l"] Oct 30 06:36:25.250599 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:36:25.250572 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ff22bf_0ec1_40d9_9b33_bfb2aad4e4dd.slice/crio-130649c018b0d9793ab5118222e29cc3555d607c407a1262aa137c1e422fd6cd WatchSource:0}: Error finding container 130649c018b0d9793ab5118222e29cc3555d607c407a1262aa137c1e422fd6cd: Status 404 returned error can't find the container with id 130649c018b0d9793ab5118222e29cc3555d607c407a1262aa137c1e422fd6cd Oct 30 06:36:25.867323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:25.867229 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" event={"ID":"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd","Type":"ContainerStarted","Data":"130649c018b0d9793ab5118222e29cc3555d607c407a1262aa137c1e422fd6cd"} Oct 30 06:36:25.909932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:25.909876 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" podStartSLOduration=1.909860118 podStartE2EDuration="1.909860118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:25.906643044 +0000 UTC m=+2599.902600057" watchObservedRunningTime="2026-10-30 06:36:25.909860118 +0000 UTC m=+2599.905817135" Oct 30 06:36:26.312165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.312128 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v"] Oct 30 06:36:26.317510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.317484 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:26.336630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.336603 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v"] Oct 30 06:36:26.418021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.417984 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8546bc59-7cw7v\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:26.418237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.418217 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8546bc59-7cw7v\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:26.418287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.418270 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnxv\" (UniqueName: \"kubernetes.io/projected/531c7fc6-dd7a-429f-ae46-8b8054339b93-kube-api-access-cnnxv\") pod \"maas-api-revoke-sub-8546bc59-7cw7v\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:26.520231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.520183 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8546bc59-7cw7v\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:26.520506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.520251 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnxv\" (UniqueName: \"kubernetes.io/projected/531c7fc6-dd7a-429f-ae46-8b8054339b93-kube-api-access-cnnxv\") pod \"maas-api-revoke-sub-8546bc59-7cw7v\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:26.520597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.520513 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8546bc59-7cw7v\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:26.521183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.521163 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8546bc59-7cw7v\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:26.522878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.522854 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8546bc59-7cw7v\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:26.543315 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.543284 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnxv\" (UniqueName: \"kubernetes.io/projected/531c7fc6-dd7a-429f-ae46-8b8054339b93-kube-api-access-cnnxv\") pod \"maas-api-revoke-sub-8546bc59-7cw7v\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:26.635327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.635281 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:26.877956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.877924 5001 generic.go:356] "Generic (PLEG): container finished" podID="16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd" containerID="63ee07ec723d5d81324795d9af1227b9b6d880b64e4799a63fc3936ffd2f792b" exitCode=0 Oct 30 06:36:26.878394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.878015 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" event={"ID":"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd","Type":"ContainerDied","Data":"63ee07ec723d5d81324795d9af1227b9b6d880b64e4799a63fc3936ffd2f792b"} Oct 30 06:36:26.966562 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:26.966530 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v"] Oct 30 06:36:26.967273 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:36:26.967231 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531c7fc6_dd7a_429f_ae46_8b8054339b93.slice/crio-d00859ab6f62f751e656b1bb74b17eeb9f2d0f4062ef030d414a76ff3f6744e3 WatchSource:0}: Error finding container d00859ab6f62f751e656b1bb74b17eeb9f2d0f4062ef030d414a76ff3f6744e3: Status 404 returned error can't find the container with id d00859ab6f62f751e656b1bb74b17eeb9f2d0f4062ef030d414a76ff3f6744e3 Oct 30 06:36:27.888513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:27.888472 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" event={"ID":"531c7fc6-dd7a-429f-ae46-8b8054339b93","Type":"ContainerStarted","Data":"f73c1e0b95b5930a206798c1112921ede25eb669739ce53c721e0806d9d45410"} Oct 30 06:36:27.888942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:27.888540 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" event={"ID":"531c7fc6-dd7a-429f-ae46-8b8054339b93","Type":"ContainerStarted","Data":"d00859ab6f62f751e656b1bb74b17eeb9f2d0f4062ef030d414a76ff3f6744e3"} Oct 30 06:36:27.935036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:27.934981 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" podStartSLOduration=1.934966364 podStartE2EDuration="1.934966364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:27.929137765 +0000 UTC m=+2601.925094776" watchObservedRunningTime="2026-10-30 06:36:27.934966364 +0000 UTC m=+2601.930923373" Oct 30 06:36:28.060977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.060954 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:28.156850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.156816 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-kube-api-access-h6xd6\" (UniqueName: \"kubernetes.io/projected/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-kube-api-access-h6xd6\") pod \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " Oct 30 06:36:28.157233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.157212 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-xks-ca\") pod \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " Oct 30 06:36:28.157456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.157439 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-openshift-ca\") pod \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\" (UID: \"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd\") " Oct 30 06:36:28.160566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.160534 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-kube-api-access-h6xd6" pod "16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd" (UID: "16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd"). InnerVolumeSpecName "kube-api-access-h6xd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:28.160690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.160596 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-openshift-ca" pod "16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd" (UID: "16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:28.170441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.170406 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-xks-ca" pod "16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd" (UID: "16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:36:28.260997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.260962 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:28.260997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.260993 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:28.260997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.261004 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6xd6\" (UniqueName: \"kubernetes.io/projected/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd-kube-api-access-h6xd6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:28.899082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.899054 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" Oct 30 06:36:28.899525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.899093 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l" event={"ID":"16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd","Type":"ContainerDied","Data":"130649c018b0d9793ab5118222e29cc3555d607c407a1262aa137c1e422fd6cd"} Oct 30 06:36:28.899525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.899171 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="130649c018b0d9793ab5118222e29cc3555d607c407a1262aa137c1e422fd6cd" Oct 30 06:36:28.901265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.901241 5001 generic.go:356] "Generic (PLEG): container finished" podID="531c7fc6-dd7a-429f-ae46-8b8054339b93" containerID="f73c1e0b95b5930a206798c1112921ede25eb669739ce53c721e0806d9d45410" exitCode=0 Oct 30 06:36:28.901430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:28.901269 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" event={"ID":"531c7fc6-dd7a-429f-ae46-8b8054339b93","Type":"ContainerDied","Data":"f73c1e0b95b5930a206798c1112921ede25eb669739ce53c721e0806d9d45410"} Oct 30 06:36:30.105628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.105598 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:30.202663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.202556 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-openshift-ca\") pod \"531c7fc6-dd7a-429f-ae46-8b8054339b93\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " Oct 30 06:36:30.202866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.202802 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-xks-ca\") pod \"531c7fc6-dd7a-429f-ae46-8b8054339b93\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " Oct 30 06:36:30.202941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.202897 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531c7fc6-dd7a-429f-ae46-8b8054339b93-kube-api-access-cnnxv\" (UniqueName: \"kubernetes.io/projected/531c7fc6-dd7a-429f-ae46-8b8054339b93-kube-api-access-cnnxv\") pod \"531c7fc6-dd7a-429f-ae46-8b8054339b93\" (UID: \"531c7fc6-dd7a-429f-ae46-8b8054339b93\") " Oct 30 06:36:30.203047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.202982 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-openshift-ca" pod "531c7fc6-dd7a-429f-ae46-8b8054339b93" (UID: "531c7fc6-dd7a-429f-ae46-8b8054339b93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:30.203867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.203843 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:30.206035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.206002 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-xks-ca" pod "531c7fc6-dd7a-429f-ae46-8b8054339b93" (UID: "531c7fc6-dd7a-429f-ae46-8b8054339b93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:36:30.206458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.206430 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531c7fc6-dd7a-429f-ae46-8b8054339b93-kube-api-access-cnnxv" pod "531c7fc6-dd7a-429f-ae46-8b8054339b93" (UID: "531c7fc6-dd7a-429f-ae46-8b8054339b93"). InnerVolumeSpecName "kube-api-access-cnnxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:30.305657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.305623 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531c7fc6-dd7a-429f-ae46-8b8054339b93-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:30.305657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.305657 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnnxv\" (UniqueName: \"kubernetes.io/projected/531c7fc6-dd7a-429f-ae46-8b8054339b93-kube-api-access-cnnxv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:30.921074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.921040 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" Oct 30 06:36:30.921074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.921068 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v" event={"ID":"531c7fc6-dd7a-429f-ae46-8b8054339b93","Type":"ContainerDied","Data":"d00859ab6f62f751e656b1bb74b17eeb9f2d0f4062ef030d414a76ff3f6744e3"} Oct 30 06:36:30.921363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:30.921115 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d00859ab6f62f751e656b1bb74b17eeb9f2d0f4062ef030d414a76ff3f6744e3" Oct 30 06:36:31.429249 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:31.426706 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj"] Oct 30 06:36:31.434977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:31.433613 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" containerID="cri-o://17a315d0616b8b5ee5ede024c521b177c661fcf326eda068238affb106cd41f4" gracePeriod=30 Oct 30 06:36:31.435818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:31.435329 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr"] Oct 30 06:36:31.436205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:31.436178 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" podUID="9516fde5-0a00-449c-8872-810fab38705b" containerName="istio-proxy" containerID="cri-o://9cd1037a59a3e9d06824efd8d59227562bb42ead2ded4614e6a61bb5af8a5410" gracePeriod=30 Oct 30 06:36:33.529578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:33.529541 5001 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:36:33.529982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:33.529597 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:36:34.021530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.021469 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq"] Oct 30 06:36:34.025336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.022802 5001 cpu_manager.go:414] "containerMap: removing container" podUID="16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd" containerName="revoke-keys" Oct 30 06:36:34.025336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.022827 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd" containerName="revoke-keys" Oct 30 06:36:34.025336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.022843 5001 cpu_manager.go:414] "containerMap: removing container" podUID="531c7fc6-dd7a-429f-ae46-8b8054339b93" containerName="revoke-keys" Oct 30 06:36:34.025336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.022850 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531c7fc6-dd7a-429f-ae46-8b8054339b93" containerName="revoke-keys" Oct 30 06:36:34.025336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.023144 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd" containerName="revoke-keys" Oct 30 06:36:34.025336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.023159 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="531c7fc6-dd7a-429f-ae46-8b8054339b93" containerName="revoke-keys" Oct 30 06:36:34.080330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.070643 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.083206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.083167 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq"] Oct 30 06:36:34.106872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.105664 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-88acce-openshift-default-dockercfg-hxq2d\"" Oct 30 06:36:34.153973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.153939 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-socket\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.154236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.154214 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-envoy\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.154462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.154440 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pmpv\" (UniqueName: \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-kube-api-access-5pmpv\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.154639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.154623 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-certs\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.154776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.154762 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/981ae7ff-0c71-468e-b045-9f7735860591-istio-podinfo\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.154978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.154952 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-data\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.163085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.163056 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-credential-socket\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.163473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.163450 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/981ae7ff-0c71-468e-b045-9f7735860591-istiod-ca-cert\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.213531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.192427 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-istio-token\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.374027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.373989 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-socket\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.374290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.374045 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-envoy\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.374290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.374094 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pmpv\" (UniqueName: \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-kube-api-access-5pmpv\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.374290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.374167 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-certs\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.374290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.374213 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/981ae7ff-0c71-468e-b045-9f7735860591-istio-podinfo\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.374290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.374287 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-data\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.374664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.374344 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-credential-socket\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.374664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.374447 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/981ae7ff-0c71-468e-b045-9f7735860591-istiod-ca-cert\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.374664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.374487 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-istio-token\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.375858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.375556 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-certs\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.375858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.375841 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-socket\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.378985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.376457 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-credential-socket\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.378985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.377014 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-data\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.378985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.377872 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/981ae7ff-0c71-468e-b045-9f7735860591-istiod-ca-cert\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.379841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.379816 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/981ae7ff-0c71-468e-b045-9f7735860591-istio-podinfo\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.382442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.382417 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-envoy\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.459231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.457444 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pmpv\" (UniqueName: \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-kube-api-access-5pmpv\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.460287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.460211 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-istio-token\") pod \"e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:34.679872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:34.679777 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:35.011390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:35.011360 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq"] Oct 30 06:36:35.012034 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:36:35.012011 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981ae7ff_0c71_468e_b045_9f7735860591.slice/crio-879b54e7b41751eed888513bfb50c5cbc4c280864aab66455d55e2710e21de2f WatchSource:0}: Error finding container 879b54e7b41751eed888513bfb50c5cbc4c280864aab66455d55e2710e21de2f: Status 404 returned error can't find the container with id 879b54e7b41751eed888513bfb50c5cbc4c280864aab66455d55e2710e21de2f Oct 30 06:36:35.014885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:35.014853 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:35.014975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:35.014917 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:35.014975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:35.014949 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:35.960238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:35.960202 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9"] Oct 30 06:36:36.000328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:35.999985 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" event={"ID":"981ae7ff-0c71-468e-b045-9f7735860591","Type":"ContainerStarted","Data":"0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc"} Oct 30 06:36:36.000328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:36.000054 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" event={"ID":"981ae7ff-0c71-468e-b045-9f7735860591","Type":"ContainerStarted","Data":"879b54e7b41751eed888513bfb50c5cbc4c280864aab66455d55e2710e21de2f"} Oct 30 06:36:36.050631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:36.050576 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" podStartSLOduration=3.050562319 podStartE2EDuration="3.050562319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:36.045744507 +0000 UTC m=+2610.041701538" watchObservedRunningTime="2026-10-30 06:36:36.050562319 +0000 UTC m=+2610.046519329" Oct 30 06:36:36.691364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:36.691289 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:36.774872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:36.768819 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:37.017430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.017398 5001 generic.go:356] "Generic (PLEG): container finished" podID="9516fde5-0a00-449c-8872-810fab38705b" containerID="9cd1037a59a3e9d06824efd8d59227562bb42ead2ded4614e6a61bb5af8a5410" exitCode=0 Oct 30 06:36:37.017955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.017682 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" event={"ID":"9516fde5-0a00-449c-8872-810fab38705b","Type":"ContainerDied","Data":"9cd1037a59a3e9d06824efd8d59227562bb42ead2ded4614e6a61bb5af8a5410"} Oct 30 06:36:37.030289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.024732 5001 generic.go:356] "Generic (PLEG): container finished" podID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerID="17a315d0616b8b5ee5ede024c521b177c661fcf326eda068238affb106cd41f4" exitCode=0 Oct 30 06:36:37.030289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.028877 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" event={"ID":"053335c2-28a1-476e-b3d8-9d8bf3c5e005","Type":"ContainerDied","Data":"17a315d0616b8b5ee5ede024c521b177c661fcf326eda068238affb106cd41f4"} Oct 30 06:36:37.030289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.028957 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:37.030289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.028993 5001 scope.go:122] "RemoveContainer" containerID="fc0690a4ef141685d5cfde802fa036ac592cedb4dbd57ef0c6085aea73046a5f" Oct 30 06:36:37.032148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.032129 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:36:37.077640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.077438 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:36:37.154946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.154486 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:36:37.166601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.165613 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-certs\") pod \"9516fde5-0a00-449c-8872-810fab38705b\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " Oct 30 06:36:37.166601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.165644 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-socket\") pod \"9516fde5-0a00-449c-8872-810fab38705b\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " Oct 30 06:36:37.166601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.165759 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-kube-api-access-rsd6f\" (UniqueName: \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-kube-api-access-rsd6f\") pod \"9516fde5-0a00-449c-8872-810fab38705b\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " Oct 30 06:36:37.166601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.165787 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-data\") pod \"9516fde5-0a00-449c-8872-810fab38705b\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " Oct 30 06:36:37.166601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.165833 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-istio-token\" (UniqueName: \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-istio-token\") pod \"9516fde5-0a00-449c-8872-810fab38705b\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " Oct 30 06:36:37.166601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.165879 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-envoy\") pod \"9516fde5-0a00-449c-8872-810fab38705b\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " Oct 30 06:36:37.166601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.165904 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-certs" pod "9516fde5-0a00-449c-8872-810fab38705b" (UID: "9516fde5-0a00-449c-8872-810fab38705b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:37.166601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.166004 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-credential-socket\") pod \"9516fde5-0a00-449c-8872-810fab38705b\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " Oct 30 06:36:37.166601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.166071 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9516fde5-0a00-449c-8872-810fab38705b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9516fde5-0a00-449c-8872-810fab38705b-istio-podinfo\") pod \"9516fde5-0a00-449c-8872-810fab38705b\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " Oct 30 06:36:37.166601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.166134 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9516fde5-0a00-449c-8872-810fab38705b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9516fde5-0a00-449c-8872-810fab38705b-istiod-ca-cert\") pod \"9516fde5-0a00-449c-8872-810fab38705b\" (UID: \"9516fde5-0a00-449c-8872-810fab38705b\") " Oct 30 06:36:37.167145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.166610 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-credential-socket" pod "9516fde5-0a00-449c-8872-810fab38705b" (UID: "9516fde5-0a00-449c-8872-810fab38705b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:37.168012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.167852 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.168012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.167888 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.168897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.168838 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-data" pod "9516fde5-0a00-449c-8872-810fab38705b" (UID: "9516fde5-0a00-449c-8872-810fab38705b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:37.169013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.168903 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-envoy" pod "9516fde5-0a00-449c-8872-810fab38705b" (UID: "9516fde5-0a00-449c-8872-810fab38705b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:37.169140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.169116 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-kube-api-access-rsd6f" pod "9516fde5-0a00-449c-8872-810fab38705b" (UID: "9516fde5-0a00-449c-8872-810fab38705b"). InnerVolumeSpecName "kube-api-access-rsd6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:37.169625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.169581 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9516fde5-0a00-449c-8872-810fab38705b-istiod-ca-cert" pod "9516fde5-0a00-449c-8872-810fab38705b" (UID: "9516fde5-0a00-449c-8872-810fab38705b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:37.169798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.169776 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-socket" pod "9516fde5-0a00-449c-8872-810fab38705b" (UID: "9516fde5-0a00-449c-8872-810fab38705b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:37.171589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.171562 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-istio-token" pod "9516fde5-0a00-449c-8872-810fab38705b" (UID: "9516fde5-0a00-449c-8872-810fab38705b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:37.171802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.171769 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9516fde5-0a00-449c-8872-810fab38705b-istio-podinfo" pod "9516fde5-0a00-449c-8872-810fab38705b" (UID: "9516fde5-0a00-449c-8872-810fab38705b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:36:37.289354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.287216 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.289354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.287250 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9516fde5-0a00-449c-8872-810fab38705b-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.289354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.287264 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9516fde5-0a00-449c-8872-810fab38705b-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.289354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.287277 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.289354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.287294 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsd6f\" (UniqueName: \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-kube-api-access-rsd6f\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.289354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.287322 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9516fde5-0a00-449c-8872-810fab38705b-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.289354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.287334 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9516fde5-0a00-449c-8872-810fab38705b-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.388598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.388565 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-kube-api-access-m4qrz\" (UniqueName: \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-kube-api-access-m4qrz\") pod \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " Oct 30 06:36:37.388598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.388610 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-data\") pod \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " Oct 30 06:36:37.388893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.388654 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-token\" (UniqueName: \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-token\") pod \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " Oct 30 06:36:37.388893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.388801 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-credential-socket\") pod \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " Oct 30 06:36:37.388893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.388867 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-socket\") pod \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " Oct 30 06:36:37.389072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.388902 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-envoy\") pod \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " Oct 30 06:36:37.389072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.389007 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-podinfo\") pod \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " Oct 30 06:36:37.389190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.389081 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-certs\") pod \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " Oct 30 06:36:37.389190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.389006 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-data" pod "053335c2-28a1-476e-b3d8-9d8bf3c5e005" (UID: "053335c2-28a1-476e-b3d8-9d8bf3c5e005"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:37.389190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.389122 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-socket" pod "053335c2-28a1-476e-b3d8-9d8bf3c5e005" (UID: "053335c2-28a1-476e-b3d8-9d8bf3c5e005"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:37.389190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.389088 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-credential-socket" pod "053335c2-28a1-476e-b3d8-9d8bf3c5e005" (UID: "053335c2-28a1-476e-b3d8-9d8bf3c5e005"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:37.389190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.389141 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istiod-ca-cert\") pod \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\" (UID: \"053335c2-28a1-476e-b3d8-9d8bf3c5e005\") " Oct 30 06:36:37.389497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.389383 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-certs" pod "053335c2-28a1-476e-b3d8-9d8bf3c5e005" (UID: "053335c2-28a1-476e-b3d8-9d8bf3c5e005"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:37.389565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.389539 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istiod-ca-cert" pod "053335c2-28a1-476e-b3d8-9d8bf3c5e005" (UID: "053335c2-28a1-476e-b3d8-9d8bf3c5e005"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:37.390145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.390031 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.390258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.390177 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.390258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.390190 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.390258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.390205 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.390258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.390219 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.391880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.391830 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-token" pod "053335c2-28a1-476e-b3d8-9d8bf3c5e005" (UID: "053335c2-28a1-476e-b3d8-9d8bf3c5e005"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:37.392005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.391934 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-podinfo" pod "053335c2-28a1-476e-b3d8-9d8bf3c5e005" (UID: "053335c2-28a1-476e-b3d8-9d8bf3c5e005"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:36:37.392087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.392062 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-kube-api-access-m4qrz" pod "053335c2-28a1-476e-b3d8-9d8bf3c5e005" (UID: "053335c2-28a1-476e-b3d8-9d8bf3c5e005"). InnerVolumeSpecName "kube-api-access-m4qrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:37.392158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.392133 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-envoy" pod "053335c2-28a1-476e-b3d8-9d8bf3c5e005" (UID: "053335c2-28a1-476e-b3d8-9d8bf3c5e005"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:37.492581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.492482 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.492581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.492510 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.492581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.492524 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4qrz\" (UniqueName: \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-kube-api-access-m4qrz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.492581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.492542 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053335c2-28a1-476e-b3d8-9d8bf3c5e005-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.776247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:37.776163 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc"] Oct 30 06:36:38.035377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:38.035253 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" Oct 30 06:36:38.035377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:38.035291 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr" event={"ID":"9516fde5-0a00-449c-8872-810fab38705b","Type":"ContainerDied","Data":"6691e95c842663bfa4949af3ef62b09eb0f999df5cfba0be7dd6f0858776e92b"} Oct 30 06:36:38.035377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:38.035366 5001 scope.go:122] "RemoveContainer" containerID="9cd1037a59a3e9d06824efd8d59227562bb42ead2ded4614e6a61bb5af8a5410" Oct 30 06:36:38.037370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:38.037298 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" Oct 30 06:36:38.037480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:38.037361 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj" event={"ID":"053335c2-28a1-476e-b3d8-9d8bf3c5e005","Type":"ContainerDied","Data":"36a46a213c154098df882e8d83010a2408940da60b0e2dfa74c3b9fd3b84aae5"} Oct 30 06:36:38.058658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:38.058637 5001 scope.go:122] "RemoveContainer" containerID="17a315d0616b8b5ee5ede024c521b177c661fcf326eda068238affb106cd41f4" Oct 30 06:36:38.521336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:38.512796 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr"] Oct 30 06:36:38.675632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:38.675595 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-qf9cr"] Oct 30 06:36:38.687607 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:36:38.687570 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:36:38.704478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:38.704406 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj"] Oct 30 06:36:38.711051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:38.711021 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-cxhmj"] Oct 30 06:36:40.120133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.120092 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" path="/var/lib/kubelet/pods/053335c2-28a1-476e-b3d8-9d8bf3c5e005/volumes" Oct 30 06:36:40.120873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.120852 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9516fde5-0a00-449c-8872-810fab38705b" path="/var/lib/kubelet/pods/9516fde5-0a00-449c-8872-810fab38705b/volumes" Oct 30 06:36:40.531855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.531779 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm"] Oct 30 06:36:40.532648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.532631 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9516fde5-0a00-449c-8872-810fab38705b" containerName="istio-proxy" Oct 30 06:36:40.532648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.532648 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9516fde5-0a00-449c-8872-810fab38705b" containerName="istio-proxy" Oct 30 06:36:40.532781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.532684 5001 cpu_manager.go:414] "containerMap: removing container" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" Oct 30 06:36:40.532781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.532690 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" Oct 30 06:36:40.532781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.532720 5001 cpu_manager.go:414] "containerMap: removing container" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" Oct 30 06:36:40.532781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.532726 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" Oct 30 06:36:40.532923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.532881 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" Oct 30 06:36:40.532923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.532893 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="053335c2-28a1-476e-b3d8-9d8bf3c5e005" containerName="istio-proxy" Oct 30 06:36:40.532923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.532905 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9516fde5-0a00-449c-8872-810fab38705b" containerName="istio-proxy" Oct 30 06:36:40.538536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.538518 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:36:40.560345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.560269 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm"] Oct 30 06:36:40.562437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.562412 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-88acce\"" Oct 30 06:36:40.562594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.562547 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-88acce-dockercfg-gnr69\"" Oct 30 06:36:40.606328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.604036 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm\" (UID: \"2e9c2db0-e1f2-4817-8bb0-90062efa45e1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:36:40.606541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.606353 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqpzt\" (UniqueName: \"kubernetes.io/projected/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-kube-api-access-kqpzt\") pod \"payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm\" (UID: \"2e9c2db0-e1f2-4817-8bb0-90062efa45e1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:36:40.782706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.782607 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpzt\" (UniqueName: \"kubernetes.io/projected/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-kube-api-access-kqpzt\") pod \"payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm\" (UID: \"2e9c2db0-e1f2-4817-8bb0-90062efa45e1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:36:40.783167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.783143 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm\" (UID: \"2e9c2db0-e1f2-4817-8bb0-90062efa45e1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:36:40.833420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.833373 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm\" (UID: \"2e9c2db0-e1f2-4817-8bb0-90062efa45e1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:36:40.978409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.964778 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" podUID="d6ad59f5-6a91-4781-9c89-7802aa3f29c3" containerName="payload-processing" containerID="cri-o://4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95" gracePeriod=25 Oct 30 06:36:40.997671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:40.996516 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpzt\" (UniqueName: \"kubernetes.io/projected/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-kube-api-access-kqpzt\") pod \"payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm\" (UID: \"2e9c2db0-e1f2-4817-8bb0-90062efa45e1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:36:41.027906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.018795 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" podUID="d6ad59f5-6a91-4781-9c89-7802aa3f29c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:36:41.170346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.162645 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:36:41.400350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.400283 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:36:41.519639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.519605 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-kube-api-access-smqlz\" (UniqueName: \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-kube-api-access-smqlz\") pod \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " Oct 30 06:36:41.519813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.519690 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-provider-credentials\") pod \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " Oct 30 06:36:41.519813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.519708 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-plugins-config-volume\") pod \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " Oct 30 06:36:41.519918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.519830 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-routing-overlay-volume\") pod \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\" (UID: \"d6ad59f5-6a91-4781-9c89-7802aa3f29c3\") " Oct 30 06:36:41.520209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.520183 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-routing-overlay-volume" pod "d6ad59f5-6a91-4781-9c89-7802aa3f29c3" (UID: "d6ad59f5-6a91-4781-9c89-7802aa3f29c3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:41.520209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.520191 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-plugins-config-volume" pod "d6ad59f5-6a91-4781-9c89-7802aa3f29c3" (UID: "d6ad59f5-6a91-4781-9c89-7802aa3f29c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:41.520679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.520660 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:41.520679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.520679 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:41.522206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.522182 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-kube-api-access-smqlz" pod "d6ad59f5-6a91-4781-9c89-7802aa3f29c3" (UID: "d6ad59f5-6a91-4781-9c89-7802aa3f29c3"). InnerVolumeSpecName "kube-api-access-smqlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:41.522325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.522215 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-provider-credentials" pod "d6ad59f5-6a91-4781-9c89-7802aa3f29c3" (UID: "d6ad59f5-6a91-4781-9c89-7802aa3f29c3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:41.544355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.544332 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm"] Oct 30 06:36:41.545862 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:36:41.545834 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9c2db0_e1f2_4817_8bb0_90062efa45e1.slice/crio-2e319cd699b82a2f4a0bd412fd5572e547fa382afd3448d5d0d6ea1b1ecdad79 WatchSource:0}: Error finding container 2e319cd699b82a2f4a0bd412fd5572e547fa382afd3448d5d0d6ea1b1ecdad79: Status 404 returned error can't find the container with id 2e319cd699b82a2f4a0bd412fd5572e547fa382afd3448d5d0d6ea1b1ecdad79 Oct 30 06:36:41.621979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.621950 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smqlz\" (UniqueName: \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-kube-api-access-smqlz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:41.621979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:41.621975 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6ad59f5-6a91-4781-9c89-7802aa3f29c3-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:42.093034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.092971 5001 generic.go:356] "Generic (PLEG): container finished" podID="d6ad59f5-6a91-4781-9c89-7802aa3f29c3" containerID="4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95" exitCode=0 Oct 30 06:36:42.093854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.093828 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" Oct 30 06:36:42.094937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.094900 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" event={"ID":"d6ad59f5-6a91-4781-9c89-7802aa3f29c3","Type":"ContainerDied","Data":"4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95"} Oct 30 06:36:42.095141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.095017 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9" event={"ID":"d6ad59f5-6a91-4781-9c89-7802aa3f29c3","Type":"ContainerDied","Data":"bcfe0797d5bc58f3de4618a00042d73e121b52fd16e20070f33a7c2d2ca1f14c"} Oct 30 06:36:42.095141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.095057 5001 scope.go:122] "RemoveContainer" containerID="4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95" Oct 30 06:36:42.097281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.096816 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" event={"ID":"2e9c2db0-e1f2-4817-8bb0-90062efa45e1","Type":"ContainerStarted","Data":"13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b"} Oct 30 06:36:42.097281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.096882 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" event={"ID":"2e9c2db0-e1f2-4817-8bb0-90062efa45e1","Type":"ContainerStarted","Data":"2e319cd699b82a2f4a0bd412fd5572e547fa382afd3448d5d0d6ea1b1ecdad79"} Oct 30 06:36:42.097281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.097217 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:36:42.117648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.117617 5001 scope.go:122] "RemoveContainer" containerID="4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95" Oct 30 06:36:42.118077 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:36:42.118039 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95\": container with ID starting with 4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95 not found: ID does not exist" containerID="4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95" Oct 30 06:36:42.118188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.118081 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95"} err="failed to get container status \"4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95\": rpc error: code = NotFound desc = could not find container \"4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95\": container with ID starting with 4dbc72aaee1ae008612b73cf3a7850a99353cd7933d03da2703d2591d9435a95 not found: ID does not exist" Oct 30 06:36:42.151803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.151752 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" podStartSLOduration=2.151738016 podStartE2EDuration="2.151738016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:42.148610592 +0000 UTC m=+2616.144567602" watchObservedRunningTime="2026-10-30 06:36:42.151738016 +0000 UTC m=+2616.147695031" Oct 30 06:36:42.189255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.189215 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9"] Oct 30 06:36:42.197538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.197506 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-7555xwsb9"] Oct 30 06:36:42.776886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:42.776843 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" podUID="9c389a4d-4f6e-4610-bf42-7de433f41ee1" containerName="payload-processing" containerID="cri-o://c3a5eb816ffb42e1cad238c73a25bfef00b74b4010571e8ae652e7e1af372734" gracePeriod=25 Oct 30 06:36:43.112276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.112241 5001 generic.go:356] "Generic (PLEG): container finished" podID="9c389a4d-4f6e-4610-bf42-7de433f41ee1" containerID="c3a5eb816ffb42e1cad238c73a25bfef00b74b4010571e8ae652e7e1af372734" exitCode=0 Oct 30 06:36:43.113160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.113135 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" event={"ID":"9c389a4d-4f6e-4610-bf42-7de433f41ee1","Type":"ContainerDied","Data":"c3a5eb816ffb42e1cad238c73a25bfef00b74b4010571e8ae652e7e1af372734"} Oct 30 06:36:43.160171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.160142 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:36:43.273686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.273646 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-provider-credentials\") pod \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " Oct 30 06:36:43.274169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.273715 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-routing-overlay-volume\") pod \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " Oct 30 06:36:43.274169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.273864 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-plugins-config-volume\") pod \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " Oct 30 06:36:43.274169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.273933 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-kube-api-access-8srj4\" (UniqueName: \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-kube-api-access-8srj4\") pod \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\" (UID: \"9c389a4d-4f6e-4610-bf42-7de433f41ee1\") " Oct 30 06:36:43.274434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.274099 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-routing-overlay-volume" pod "9c389a4d-4f6e-4610-bf42-7de433f41ee1" (UID: "9c389a4d-4f6e-4610-bf42-7de433f41ee1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:43.274434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.274351 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-plugins-config-volume" pod "9c389a4d-4f6e-4610-bf42-7de433f41ee1" (UID: "9c389a4d-4f6e-4610-bf42-7de433f41ee1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:43.275374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.275348 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:43.275532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.275392 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c389a4d-4f6e-4610-bf42-7de433f41ee1-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:43.276546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.276523 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-provider-credentials" pod "9c389a4d-4f6e-4610-bf42-7de433f41ee1" (UID: "9c389a4d-4f6e-4610-bf42-7de433f41ee1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:43.276883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.276860 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-kube-api-access-8srj4" pod "9c389a4d-4f6e-4610-bf42-7de433f41ee1" (UID: "9c389a4d-4f6e-4610-bf42-7de433f41ee1"). InnerVolumeSpecName "kube-api-access-8srj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:43.286826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.286798 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:36:43.377436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.377358 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:43.377436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:43.377392 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8srj4\" (UniqueName: \"kubernetes.io/projected/9c389a4d-4f6e-4610-bf42-7de433f41ee1-kube-api-access-8srj4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:36:44.120820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:44.120783 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ad59f5-6a91-4781-9c89-7802aa3f29c3" path="/var/lib/kubelet/pods/d6ad59f5-6a91-4781-9c89-7802aa3f29c3/volumes" Oct 30 06:36:44.123882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:44.123847 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" Oct 30 06:36:44.124034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:44.123871 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc" event={"ID":"9c389a4d-4f6e-4610-bf42-7de433f41ee1","Type":"ContainerDied","Data":"0b90452dd965a28698391f25313c4f953ef89aea378a61e0e7b97d25d7f11936"} Oct 30 06:36:44.124034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:44.123952 5001 scope.go:122] "RemoveContainer" containerID="c3a5eb816ffb42e1cad238c73a25bfef00b74b4010571e8ae652e7e1af372734" Oct 30 06:36:44.177200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:44.177167 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc"] Oct 30 06:36:44.188155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:44.188126 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-69f3fa/payload-processing-external-model-e2e-extmod-b-69f3fa-86d9cn4lc"] Oct 30 06:36:46.122877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:46.122845 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c389a4d-4f6e-4610-bf42-7de433f41ee1" path="/var/lib/kubelet/pods/9c389a4d-4f6e-4610-bf42-7de433f41ee1/volumes" Oct 30 06:36:48.873499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.873466 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9"] Oct 30 06:36:48.874285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.874269 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d6ad59f5-6a91-4781-9c89-7802aa3f29c3" containerName="payload-processing" Oct 30 06:36:48.874285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.874285 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ad59f5-6a91-4781-9c89-7802aa3f29c3" containerName="payload-processing" Oct 30 06:36:48.874491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.874319 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9c389a4d-4f6e-4610-bf42-7de433f41ee1" containerName="payload-processing" Oct 30 06:36:48.874491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.874324 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c389a4d-4f6e-4610-bf42-7de433f41ee1" containerName="payload-processing" Oct 30 06:36:48.874491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.874469 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ad59f5-6a91-4781-9c89-7802aa3f29c3" containerName="payload-processing" Oct 30 06:36:48.874491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.874480 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c389a4d-4f6e-4610-bf42-7de433f41ee1" containerName="payload-processing" Oct 30 06:36:48.879920 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.879900 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:48.899722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.899690 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9"] Oct 30 06:36:48.904311 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.904279 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"agc-cred-9523b214660a\"" Oct 30 06:36:48.956827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.956789 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snsvv\" (UniqueName: \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-kube-api-access-snsvv\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:48.957022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.956948 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:48.957069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.957051 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:48.957214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:48.957199 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:49.059464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:49.059426 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snsvv\" (UniqueName: \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-kube-api-access-snsvv\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:49.059702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:49.059543 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:49.059702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:49.059642 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:49.059854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:49.059778 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:49.060593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:49.060563 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:49.060593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:49.060581 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:49.062921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:49.062897 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:49.082924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:49.082895 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snsvv\" (UniqueName: \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-kube-api-access-snsvv\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:49.200627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:49.200535 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:49.521375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:49.521277 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9"] Oct 30 06:36:49.522411 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:36:49.522379 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f4b82a_51f3_4a04_b010_c56360ae9ad0.slice/crio-2fb1ae8e32eb974cab99e6190b603ae5fa28a191bbde893f5a17d410f143259b WatchSource:0}: Error finding container 2fb1ae8e32eb974cab99e6190b603ae5fa28a191bbde893f5a17d410f143259b: Status 404 returned error can't find the container with id 2fb1ae8e32eb974cab99e6190b603ae5fa28a191bbde893f5a17d410f143259b Oct 30 06:36:50.202895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:50.202857 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" event={"ID":"66f4b82a-51f3-4a04-b010-c56360ae9ad0","Type":"ContainerStarted","Data":"a5f85444e072b3b2dc247301ad29b17966cd43005766cd66425b914f96722c4f"} Oct 30 06:36:50.203355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:50.202922 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" event={"ID":"66f4b82a-51f3-4a04-b010-c56360ae9ad0","Type":"ContainerStarted","Data":"2fb1ae8e32eb974cab99e6190b603ae5fa28a191bbde893f5a17d410f143259b"} Oct 30 06:36:50.203355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:50.202953 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:36:50.248395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:50.248344 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" podStartSLOduration=2.24832922 podStartE2EDuration="2.24832922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:50.242552811 +0000 UTC m=+2624.238509821" watchObservedRunningTime="2026-10-30 06:36:50.24832922 +0000 UTC m=+2624.244286231" Oct 30 06:36:53.034327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.034100 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28"] Oct 30 06:36:53.054136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.054104 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.075105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.075070 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28"] Oct 30 06:36:53.090480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.090124 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"e2e-cmp-legacy-7f7ea8-provider-cred\"" Oct 30 06:36:53.090480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.090183 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2d2923\"" Oct 30 06:36:53.090480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.090215 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"routing-overlay\"" Oct 30 06:36:53.090480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.090253 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"openshift-service-ca.crt\"" Oct 30 06:36:53.090480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.090324 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"kube-root-ca.crt\"" Oct 30 06:36:53.090480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.090420 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"payload-processing-external-model-e2e--d83828a5-dockercfg-qv27d\"" Oct 30 06:36:53.116506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.116482 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:36:53.190355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.190273 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.191141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.191114 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.191473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.191453 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6pf8\" (UniqueName: \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-kube-api-access-v6pf8\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.191793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.191768 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.297682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.297417 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.298165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.298122 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.298285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.298231 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6pf8\" (UniqueName: \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-kube-api-access-v6pf8\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.298451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.298431 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.301695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.299476 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.302696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.302670 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.337627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.336789 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.353236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.353202 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6pf8\" (UniqueName: \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-kube-api-access-v6pf8\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.394648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.394610 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:53.960529 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:36:53.960486 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf1e732_da1b_4560_a14f_970c9d19a8ff.slice/crio-22eda8d9d0c9b992d1be5a5a95e56e0e336dd4c8c68aa63a03b368278a7598f3 WatchSource:0}: Error finding container 22eda8d9d0c9b992d1be5a5a95e56e0e336dd4c8c68aa63a03b368278a7598f3: Status 404 returned error can't find the container with id 22eda8d9d0c9b992d1be5a5a95e56e0e336dd4c8c68aa63a03b368278a7598f3 Oct 30 06:36:53.962105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:53.962075 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28"] Oct 30 06:36:54.243869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:54.243786 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" event={"ID":"fdf1e732-da1b-4560-a14f-970c9d19a8ff","Type":"ContainerStarted","Data":"315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27"} Oct 30 06:36:54.243869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:54.243834 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" event={"ID":"fdf1e732-da1b-4560-a14f-970c9d19a8ff","Type":"ContainerStarted","Data":"22eda8d9d0c9b992d1be5a5a95e56e0e336dd4c8c68aa63a03b368278a7598f3"} Oct 30 06:36:54.244595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:54.243923 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:36:54.289721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:54.289659 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" podStartSLOduration=2.289639969 podStartE2EDuration="2.289639969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:54.282959937 +0000 UTC m=+2628.278916951" watchObservedRunningTime="2026-10-30 06:36:54.289639969 +0000 UTC m=+2628.285596982" Oct 30 06:36:55.740578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.740520 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5"] Oct 30 06:36:55.747484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.747459 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.755117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.755089 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5"] Oct 30 06:36:55.773826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.773790 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"routing-overlay\"" Oct 30 06:36:55.773826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.773798 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b8ca49\"" Oct 30 06:36:55.774054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.773798 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"e2e-cmp-legacy-450f2d-provider-cred\"" Oct 30 06:36:55.774054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.773873 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"payload-processing-external-model-e2e--7e273d00-dockercfg-g2q65\"" Oct 30 06:36:55.774054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.773896 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"kube-root-ca.crt\"" Oct 30 06:36:55.774054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.773992 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"openshift-service-ca.crt\"" Oct 30 06:36:55.834379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.834337 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrgzc\" (UniqueName: \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-kube-api-access-lrgzc\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.834581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.834468 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.834633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.834620 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.834829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.834803 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.940331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.939410 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.940331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.939527 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrgzc\" (UniqueName: \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-kube-api-access-lrgzc\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.940331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.939609 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.940331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.939787 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.945326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.942135 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.945326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.942619 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.977332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.975214 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:55.977332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:55.975908 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrgzc\" (UniqueName: \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-kube-api-access-lrgzc\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:56.079382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:56.077619 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:56.615393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:56.615356 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5"] Oct 30 06:36:56.616546 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:36:56.616519 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee85c60_c718_4b79_a952_14f256ca8be9.slice/crio-1ba94b96be5ccb58bf08bddd2a2c09bcd5e7dfd89b5b9ab5c427ad006b9d8191 WatchSource:0}: Error finding container 1ba94b96be5ccb58bf08bddd2a2c09bcd5e7dfd89b5b9ab5c427ad006b9d8191: Status 404 returned error can't find the container with id 1ba94b96be5ccb58bf08bddd2a2c09bcd5e7dfd89b5b9ab5c427ad006b9d8191 Oct 30 06:36:57.334189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:57.334155 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" event={"ID":"6ee85c60-c718-4b79-a952-14f256ca8be9","Type":"ContainerStarted","Data":"e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b"} Oct 30 06:36:57.334693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:57.334212 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" event={"ID":"6ee85c60-c718-4b79-a952-14f256ca8be9","Type":"ContainerStarted","Data":"1ba94b96be5ccb58bf08bddd2a2c09bcd5e7dfd89b5b9ab5c427ad006b9d8191"} Oct 30 06:36:57.334693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:57.334266 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:36:57.375458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:36:57.375408 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" podStartSLOduration=2.375394768 podStartE2EDuration="2.375394768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:57.373142623 +0000 UTC m=+2631.369099633" watchObservedRunningTime="2026-10-30 06:36:57.375394768 +0000 UTC m=+2631.371351778" Oct 30 06:37:01.216676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:01.216640 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:37:01.307033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:01.305830 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9"] Oct 30 06:37:05.256156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:05.256127 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:37:06.306298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.306263 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" podUID="c03fe03b-18ae-4f95-b729-09f0a9af4f58" containerName="payload-processing" containerID="cri-o://88ea7b36e11f03994164c0baee1c47f2044249e06923a707665914c77488e50d" gracePeriod=25 Oct 30 06:37:06.467432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.467396 5001 generic.go:356] "Generic (PLEG): container finished" podID="c03fe03b-18ae-4f95-b729-09f0a9af4f58" containerID="88ea7b36e11f03994164c0baee1c47f2044249e06923a707665914c77488e50d" exitCode=0 Oct 30 06:37:06.467619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.467479 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" event={"ID":"c03fe03b-18ae-4f95-b729-09f0a9af4f58","Type":"ContainerDied","Data":"88ea7b36e11f03994164c0baee1c47f2044249e06923a707665914c77488e50d"} Oct 30 06:37:06.617014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.616983 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:37:06.793136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.793106 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-plugins-config-volume\") pod \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " Oct 30 06:37:06.793364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.793256 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-provider-credentials\") pod \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " Oct 30 06:37:06.793486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.793466 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-kube-api-access-t65gg\" (UniqueName: \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-kube-api-access-t65gg\") pod \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " Oct 30 06:37:06.793550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.793509 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-plugins-config-volume" pod "c03fe03b-18ae-4f95-b729-09f0a9af4f58" (UID: "c03fe03b-18ae-4f95-b729-09f0a9af4f58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:06.793712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.793690 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-routing-overlay-volume\") pod \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\" (UID: \"c03fe03b-18ae-4f95-b729-09f0a9af4f58\") " Oct 30 06:37:06.794161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.794136 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-routing-overlay-volume" pod "c03fe03b-18ae-4f95-b729-09f0a9af4f58" (UID: "c03fe03b-18ae-4f95-b729-09f0a9af4f58"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:06.794958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.794932 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:06.795049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.794966 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03fe03b-18ae-4f95-b729-09f0a9af4f58-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:06.795794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.795772 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-provider-credentials" pod "c03fe03b-18ae-4f95-b729-09f0a9af4f58" (UID: "c03fe03b-18ae-4f95-b729-09f0a9af4f58"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:06.796019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.795999 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-kube-api-access-t65gg" pod "c03fe03b-18ae-4f95-b729-09f0a9af4f58" (UID: "c03fe03b-18ae-4f95-b729-09f0a9af4f58"). InnerVolumeSpecName "kube-api-access-t65gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:06.896763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.896722 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:06.896763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:06.896752 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t65gg\" (UniqueName: \"kubernetes.io/projected/c03fe03b-18ae-4f95-b729-09f0a9af4f58-kube-api-access-t65gg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:07.479522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:07.479485 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" Oct 30 06:37:07.480070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:07.479513 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9" event={"ID":"c03fe03b-18ae-4f95-b729-09f0a9af4f58","Type":"ContainerDied","Data":"a56736d9195c64064aad1a196df44cf7312bd4441dcee12d06e13f159b416ff7"} Oct 30 06:37:07.480070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:07.479585 5001 scope.go:122] "RemoveContainer" containerID="88ea7b36e11f03994164c0baee1c47f2044249e06923a707665914c77488e50d" Oct 30 06:37:07.536437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:07.536405 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9"] Oct 30 06:37:07.536817 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:37:07.536788 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03fe03b_18ae_4f95_b729_09f0a9af4f58.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03fe03b_18ae_4f95_b729_09f0a9af4f58.slice/crio-a56736d9195c64064aad1a196df44cf7312bd4441dcee12d06e13f159b416ff7\": RecentStats: unable to find data in memory cache]" Oct 30 06:37:07.536877 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:37:07.536840 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03fe03b_18ae_4f95_b729_09f0a9af4f58.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03fe03b_18ae_4f95_b729_09f0a9af4f58.slice/crio-a56736d9195c64064aad1a196df44cf7312bd4441dcee12d06e13f159b416ff7\": RecentStats: unable to find data in memory cache]" Oct 30 06:37:07.547867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:07.547843 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b7qnr9"] Oct 30 06:37:08.119521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:08.119487 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03fe03b-18ae-4f95-b729-09f0a9af4f58" path="/var/lib/kubelet/pods/c03fe03b-18ae-4f95-b729-09f0a9af4f58/volumes" Oct 30 06:37:08.267920 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:37:08.267876 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:37:08.346713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:08.346688 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:37:10.033452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:10.033415 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw"] Oct 30 06:37:10.044852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:10.044825 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be67b9d1-s6flw"] Oct 30 06:37:10.049717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:10.049701 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg"] Oct 30 06:37:10.054396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:10.054371 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10a218b-9slwg"] Oct 30 06:37:10.120011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:10.119981 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ead8c6-41c8-40e6-9fe7-6dedaaeab73f" path="/var/lib/kubelet/pods/43ead8c6-41c8-40e6-9fe7-6dedaaeab73f/volumes" Oct 30 06:37:10.120490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:10.120474 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592b6f9a-cfd5-4f7f-a50a-3da42e914750" path="/var/lib/kubelet/pods/592b6f9a-cfd5-4f7f-a50a-3da42e914750/volumes" Oct 30 06:37:10.253880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:10.253846 5001 scope.go:122] "RemoveContainer" containerID="a37221e0c28844fddc124bc72338798ddb99b6a517ca6b1b3a715b0a0c2ed6c8" Oct 30 06:37:10.267637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:10.267614 5001 scope.go:122] "RemoveContainer" containerID="02e56a8ef754a00af780c63ec18483aed1010bb5b1a9fc0db76801b289d95bea" Oct 30 06:37:18.498964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.498868 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr"] Oct 30 06:37:18.499785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.499769 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c03fe03b-18ae-4f95-b729-09f0a9af4f58" containerName="payload-processing" Oct 30 06:37:18.499918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.499787 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03fe03b-18ae-4f95-b729-09f0a9af4f58" containerName="payload-processing" Oct 30 06:37:18.499981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.499956 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03fe03b-18ae-4f95-b729-09f0a9af4f58" containerName="payload-processing" Oct 30 06:37:18.505799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.505781 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.520337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.520278 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"kube-root-ca.crt\"" Oct 30 06:37:18.523280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.523256 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr"] Oct 30 06:37:18.528746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.528724 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"payload-processing-external-model-e2e--7b857ee5-dockercfg-m58tc\"" Oct 30 06:37:18.528876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.528757 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"routing-overlay\"" Oct 30 06:37:18.528876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.528783 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"agc-cred-f1c6610cfede\"" Oct 30 06:37:18.621225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.621188 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f0145f\"" Oct 30 06:37:18.621452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.621255 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"openshift-service-ca.crt\"" Oct 30 06:37:18.649343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.649282 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.649531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.649486 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.649626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.649610 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88j92\" (UniqueName: \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-kube-api-access-88j92\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.649684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.649669 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.750975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.750892 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.751145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.751058 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.751197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.751184 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88j92\" (UniqueName: \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-kube-api-access-88j92\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.751319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.751289 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.751727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.751700 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.752315 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.752277 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.754025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.754001 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.774558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.774526 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88j92\" (UniqueName: \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-kube-api-access-88j92\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:18.822546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:18.822513 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:19.189331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.184770 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf"] Oct 30 06:37:19.364332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.352902 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.398377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.391591 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"agc-cred-055e4daca375\"" Oct 30 06:37:19.398377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.392480 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv7xf\" (UniqueName: \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-kube-api-access-kv7xf\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.398377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.392586 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.398377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.392869 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.398377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.392933 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.417793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.417753 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf"] Oct 30 06:37:19.423913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.423882 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr"] Oct 30 06:37:19.494509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.494473 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.494663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.494543 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.494663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.494616 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv7xf\" (UniqueName: \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-kube-api-access-kv7xf\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.494791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.494700 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.495379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.495356 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.496274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.496250 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.500680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.500659 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.518097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.518060 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv7xf\" (UniqueName: \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-kube-api-access-kv7xf\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:19.740666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:19.740576 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:20.037546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:20.037461 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2"] Oct 30 06:37:20.276957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:20.274652 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz"] Oct 30 06:37:20.276957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:20.274720 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11a10a3-7ftt2"] Oct 30 06:37:20.276957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:20.274749 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83ba65d-l8rlz"] Oct 30 06:37:20.281787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:20.277898 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" event={"ID":"9de305bc-d34b-44c9-9448-b1280f98fead","Type":"ContainerStarted","Data":"2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19"} Oct 30 06:37:20.281787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:20.277973 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" event={"ID":"9de305bc-d34b-44c9-9448-b1280f98fead","Type":"ContainerStarted","Data":"91e37a9be4c5a5076dcb7ae2df47086905c6091c4e23dbb41354736020af86c4"} Oct 30 06:37:20.281787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:20.278250 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:20.283895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:20.283835 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" event={"ID":"23629bab-569f-4784-a3f5-b74113564d2e","Type":"ContainerStarted","Data":"ce2ddb3879568bd03fb68a5fced923df9e5f239737cbf029113e532d0f226f22"} Oct 30 06:37:20.287734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:20.287657 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf"] Oct 30 06:37:20.323503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:20.323455 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" podStartSLOduration=2.323438468 podStartE2EDuration="2.323438468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:37:20.317627969 +0000 UTC m=+2654.313584979" watchObservedRunningTime="2026-10-30 06:37:20.323438468 +0000 UTC m=+2654.319395826" Oct 30 06:37:21.308382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:21.296660 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" event={"ID":"23629bab-569f-4784-a3f5-b74113564d2e","Type":"ContainerStarted","Data":"dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3"} Oct 30 06:37:21.341220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:21.327800 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:21.373959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:21.373902 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" podStartSLOduration=3.373887661 podStartE2EDuration="3.373887661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:37:21.369089303 +0000 UTC m=+2655.365046312" watchObservedRunningTime="2026-10-30 06:37:21.373887661 +0000 UTC m=+2655.369844671" Oct 30 06:37:22.142169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:22.142131 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19764304-b00c-446e-9b9c-b3d30e8beebf" path="/var/lib/kubelet/pods/19764304-b00c-446e-9b9c-b3d30e8beebf/volumes" Oct 30 06:37:22.142820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:22.142790 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6edc91af-0a89-46a6-bd4c-cbbb4aee2336" path="/var/lib/kubelet/pods/6edc91af-0a89-46a6-bd4c-cbbb4aee2336/volumes" Oct 30 06:37:26.456086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:26.456052 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr"] Oct 30 06:37:26.459538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:26.459511 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:31.457276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.457226 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" podUID="9de305bc-d34b-44c9-9448-b1280f98fead" containerName="payload-processing" containerID="cri-o://2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19" gracePeriod=25 Oct 30 06:37:31.831355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.827823 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:31.941392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.941360 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-plugins-config-volume\") pod \"9de305bc-d34b-44c9-9448-b1280f98fead\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " Oct 30 06:37:31.941581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.941501 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-provider-credentials\") pod \"9de305bc-d34b-44c9-9448-b1280f98fead\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " Oct 30 06:37:31.941635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.941595 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-routing-overlay-volume\") pod \"9de305bc-d34b-44c9-9448-b1280f98fead\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " Oct 30 06:37:31.941721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.941688 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-kube-api-access-88j92\" (UniqueName: \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-kube-api-access-88j92\") pod \"9de305bc-d34b-44c9-9448-b1280f98fead\" (UID: \"9de305bc-d34b-44c9-9448-b1280f98fead\") " Oct 30 06:37:31.941878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.941846 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-plugins-config-volume" pod "9de305bc-d34b-44c9-9448-b1280f98fead" (UID: "9de305bc-d34b-44c9-9448-b1280f98fead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:31.942408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.942380 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-routing-overlay-volume" pod "9de305bc-d34b-44c9-9448-b1280f98fead" (UID: "9de305bc-d34b-44c9-9448-b1280f98fead"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:31.942736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.942707 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:31.942922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.942741 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de305bc-d34b-44c9-9448-b1280f98fead-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:31.944553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.944524 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-kube-api-access-88j92" pod "9de305bc-d34b-44c9-9448-b1280f98fead" (UID: "9de305bc-d34b-44c9-9448-b1280f98fead"). InnerVolumeSpecName "kube-api-access-88j92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:31.945446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:31.945422 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-provider-credentials" pod "9de305bc-d34b-44c9-9448-b1280f98fead" (UID: "9de305bc-d34b-44c9-9448-b1280f98fead"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:32.044040 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.043944 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:32.044040 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.043974 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88j92\" (UniqueName: \"kubernetes.io/projected/9de305bc-d34b-44c9-9448-b1280f98fead-kube-api-access-88j92\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:32.334791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.334716 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:37:32.417751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.417710 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9"] Oct 30 06:37:32.477151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.477114 5001 generic.go:356] "Generic (PLEG): container finished" podID="9de305bc-d34b-44c9-9448-b1280f98fead" containerID="2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19" exitCode=0 Oct 30 06:37:32.477710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.477164 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" Oct 30 06:37:32.477710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.477191 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" event={"ID":"9de305bc-d34b-44c9-9448-b1280f98fead","Type":"ContainerDied","Data":"2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19"} Oct 30 06:37:32.477710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.477253 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr" event={"ID":"9de305bc-d34b-44c9-9448-b1280f98fead","Type":"ContainerDied","Data":"91e37a9be4c5a5076dcb7ae2df47086905c6091c4e23dbb41354736020af86c4"} Oct 30 06:37:32.477710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.477284 5001 scope.go:122] "RemoveContainer" containerID="2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19" Oct 30 06:37:32.491131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.491113 5001 scope.go:122] "RemoveContainer" containerID="2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19" Oct 30 06:37:32.491454 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:37:32.491429 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19\": container with ID starting with 2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19 not found: ID does not exist" containerID="2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19" Oct 30 06:37:32.491560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.491462 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19"} err="failed to get container status \"2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19\": rpc error: code = NotFound desc = could not find container \"2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19\": container with ID starting with 2736ef19ac730948655ad2b48b2222c8c60d2d95c1ed6b23b9fb500c2372ba19 not found: ID does not exist" Oct 30 06:37:32.538788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.538736 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr"] Oct 30 06:37:32.544856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:32.544825 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-6f4c6zwtr"] Oct 30 06:37:34.119173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.119140 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de305bc-d34b-44c9-9448-b1280f98fead" path="/var/lib/kubelet/pods/9de305bc-d34b-44c9-9448-b1280f98fead/volumes" Oct 30 06:37:34.402583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.402498 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv"] Oct 30 06:37:34.403440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.403421 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9de305bc-d34b-44c9-9448-b1280f98fead" containerName="payload-processing" Oct 30 06:37:34.403440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.403439 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de305bc-d34b-44c9-9448-b1280f98fead" containerName="payload-processing" Oct 30 06:37:34.403625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.403608 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de305bc-d34b-44c9-9448-b1280f98fead" containerName="payload-processing" Oct 30 06:37:34.407734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.407716 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:34.425586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.425558 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv"] Oct 30 06:37:34.432355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.432291 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:37:34.432482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.432315 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:37:34.432482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.432401 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:37:34.435967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.435943 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:37:34.520992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.520954 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ngd\" (UniqueName: \"kubernetes.io/projected/6b11efe0-86be-4e43-994c-06ed54464099-kube-api-access-z6ngd\") pod \"maas-api-revoke-sub-9063d552-vmjwv\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:34.521195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.521123 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9063d552-vmjwv\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:34.521240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.521207 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9063d552-vmjwv\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:34.623081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.623050 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ngd\" (UniqueName: \"kubernetes.io/projected/6b11efe0-86be-4e43-994c-06ed54464099-kube-api-access-z6ngd\") pod \"maas-api-revoke-sub-9063d552-vmjwv\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:34.623276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.623156 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9063d552-vmjwv\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:34.623276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.623233 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9063d552-vmjwv\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:34.623868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.623844 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9063d552-vmjwv\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:34.625763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.625741 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9063d552-vmjwv\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:34.647759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.647734 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ngd\" (UniqueName: \"kubernetes.io/projected/6b11efe0-86be-4e43-994c-06ed54464099-kube-api-access-z6ngd\") pod \"maas-api-revoke-sub-9063d552-vmjwv\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:34.724483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:34.724398 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:35.047051 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:37:35.047012 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b11efe0_86be_4e43_994c_06ed54464099.slice/crio-23b95bcfa08b9363d94200e1820e6c3e4e0371f44f8b964756c1899aca2b8078 WatchSource:0}: Error finding container 23b95bcfa08b9363d94200e1820e6c3e4e0371f44f8b964756c1899aca2b8078: Status 404 returned error can't find the container with id 23b95bcfa08b9363d94200e1820e6c3e4e0371f44f8b964756c1899aca2b8078 Oct 30 06:37:35.048589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:35.048563 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv"] Oct 30 06:37:35.530823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:35.530790 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" event={"ID":"6b11efe0-86be-4e43-994c-06ed54464099","Type":"ContainerStarted","Data":"23b95bcfa08b9363d94200e1820e6c3e4e0371f44f8b964756c1899aca2b8078"} Oct 30 06:37:36.542113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:36.542025 5001 generic.go:356] "Generic (PLEG): container finished" podID="6b11efe0-86be-4e43-994c-06ed54464099" containerID="fd57256619446f2d7bd15580465daad0f712b944ffdff6dffc9a5edabb0cc35e" exitCode=0 Oct 30 06:37:36.542520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:36.542105 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" event={"ID":"6b11efe0-86be-4e43-994c-06ed54464099","Type":"ContainerDied","Data":"fd57256619446f2d7bd15580465daad0f712b944ffdff6dffc9a5edabb0cc35e"} Oct 30 06:37:36.706334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:36.705477 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:37:37.419113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:37.419061 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" podUID="66f4b82a-51f3-4a04-b010-c56360ae9ad0" containerName="payload-processing" containerID="cri-o://a5f85444e072b3b2dc247301ad29b17966cd43005766cd66425b914f96722c4f" gracePeriod=25 Oct 30 06:37:37.679153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:37.679066 5001 generic.go:356] "Generic (PLEG): container finished" podID="66f4b82a-51f3-4a04-b010-c56360ae9ad0" containerID="a5f85444e072b3b2dc247301ad29b17966cd43005766cd66425b914f96722c4f" exitCode=0 Oct 30 06:37:37.680062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:37.679707 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" event={"ID":"66f4b82a-51f3-4a04-b010-c56360ae9ad0","Type":"ContainerDied","Data":"a5f85444e072b3b2dc247301ad29b17966cd43005766cd66425b914f96722c4f"} Oct 30 06:37:39.113501 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:37:39.112335 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:37:39.368388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.363273 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:37:39.461845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.461741 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:39.514047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.514006 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-routing-overlay-volume\") pod \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " Oct 30 06:37:39.514243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.514139 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-provider-credentials\") pod \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " Oct 30 06:37:39.514334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.514253 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-plugins-config-volume\") pod \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " Oct 30 06:37:39.514399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.514357 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-kube-api-access-snsvv\" (UniqueName: \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-kube-api-access-snsvv\") pod \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\" (UID: \"66f4b82a-51f3-4a04-b010-c56360ae9ad0\") " Oct 30 06:37:39.516418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.516390 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-plugins-config-volume" pod "66f4b82a-51f3-4a04-b010-c56360ae9ad0" (UID: "66f4b82a-51f3-4a04-b010-c56360ae9ad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:39.516972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.516951 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-routing-overlay-volume" pod "66f4b82a-51f3-4a04-b010-c56360ae9ad0" (UID: "66f4b82a-51f3-4a04-b010-c56360ae9ad0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:39.530277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.530233 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-kube-api-access-snsvv" pod "66f4b82a-51f3-4a04-b010-c56360ae9ad0" (UID: "66f4b82a-51f3-4a04-b010-c56360ae9ad0"). InnerVolumeSpecName "kube-api-access-snsvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:39.530609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.530578 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-provider-credentials" pod "66f4b82a-51f3-4a04-b010-c56360ae9ad0" (UID: "66f4b82a-51f3-4a04-b010-c56360ae9ad0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:39.616224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.615675 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-xks-ca\") pod \"6b11efe0-86be-4e43-994c-06ed54464099\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " Oct 30 06:37:39.616224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.615939 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b11efe0-86be-4e43-994c-06ed54464099-kube-api-access-z6ngd\" (UniqueName: \"kubernetes.io/projected/6b11efe0-86be-4e43-994c-06ed54464099-kube-api-access-z6ngd\") pod \"6b11efe0-86be-4e43-994c-06ed54464099\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " Oct 30 06:37:39.616224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.616002 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-openshift-ca\") pod \"6b11efe0-86be-4e43-994c-06ed54464099\" (UID: \"6b11efe0-86be-4e43-994c-06ed54464099\") " Oct 30 06:37:39.617375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.617346 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snsvv\" (UniqueName: \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-kube-api-access-snsvv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:39.617493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.617379 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:39.617493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.617394 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66f4b82a-51f3-4a04-b010-c56360ae9ad0-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:39.617493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.617410 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f4b82a-51f3-4a04-b010-c56360ae9ad0-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:39.617759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.617739 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-openshift-ca" pod "6b11efe0-86be-4e43-994c-06ed54464099" (UID: "6b11efe0-86be-4e43-994c-06ed54464099"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:39.620329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.620234 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-xks-ca" pod "6b11efe0-86be-4e43-994c-06ed54464099" (UID: "6b11efe0-86be-4e43-994c-06ed54464099"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:37:39.621878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.621852 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b11efe0-86be-4e43-994c-06ed54464099-kube-api-access-z6ngd" pod "6b11efe0-86be-4e43-994c-06ed54464099" (UID: "6b11efe0-86be-4e43-994c-06ed54464099"). InnerVolumeSpecName "kube-api-access-z6ngd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:39.723923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.723874 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6ngd\" (UniqueName: \"kubernetes.io/projected/6b11efe0-86be-4e43-994c-06ed54464099-kube-api-access-z6ngd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:39.723923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.723926 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:39.724232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:39.723943 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b11efe0-86be-4e43-994c-06ed54464099-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:37:40.405461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:40.405364 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" event={"ID":"6b11efe0-86be-4e43-994c-06ed54464099","Type":"ContainerDied","Data":"23b95bcfa08b9363d94200e1820e6c3e4e0371f44f8b964756c1899aca2b8078"} Oct 30 06:37:40.405461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:40.405436 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b95bcfa08b9363d94200e1820e6c3e4e0371f44f8b964756c1899aca2b8078" Oct 30 06:37:40.405461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:40.405379 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv" Oct 30 06:37:40.407661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:40.407634 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" Oct 30 06:37:40.407820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:40.407635 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9" event={"ID":"66f4b82a-51f3-4a04-b010-c56360ae9ad0","Type":"ContainerDied","Data":"2fb1ae8e32eb974cab99e6190b603ae5fa28a191bbde893f5a17d410f143259b"} Oct 30 06:37:40.407820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:40.407787 5001 scope.go:122] "RemoveContainer" containerID="a5f85444e072b3b2dc247301ad29b17966cd43005766cd66425b914f96722c4f" Oct 30 06:37:40.485969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:40.485935 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9"] Oct 30 06:37:40.495704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:40.495674 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7w68v9"] Oct 30 06:37:42.123278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:42.123228 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f4b82a-51f3-4a04-b010-c56360ae9ad0" path="/var/lib/kubelet/pods/66f4b82a-51f3-4a04-b010-c56360ae9ad0/volumes" Oct 30 06:37:55.036248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:55.036209 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk"] Oct 30 06:37:55.048638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:55.048610 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c332a5-rcshk"] Oct 30 06:37:56.121926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:37:56.121887 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="434c741e-6f54-4d58-ac23-66ab4663e540" path="/var/lib/kubelet/pods/434c741e-6f54-4d58-ac23-66ab4663e540/volumes" Oct 30 06:38:02.092339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.091331 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h"] Oct 30 06:38:02.092881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.092705 5001 cpu_manager.go:414] "containerMap: removing container" podUID="66f4b82a-51f3-4a04-b010-c56360ae9ad0" containerName="payload-processing" Oct 30 06:38:02.092881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.092728 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f4b82a-51f3-4a04-b010-c56360ae9ad0" containerName="payload-processing" Oct 30 06:38:02.092881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.092783 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6b11efe0-86be-4e43-994c-06ed54464099" containerName="revoke-keys" Oct 30 06:38:02.092881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.092792 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b11efe0-86be-4e43-994c-06ed54464099" containerName="revoke-keys" Oct 30 06:38:02.093154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.093001 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f4b82a-51f3-4a04-b010-c56360ae9ad0" containerName="payload-processing" Oct 30 06:38:02.093154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.093023 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b11efe0-86be-4e43-994c-06ed54464099" containerName="revoke-keys" Oct 30 06:38:02.103810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.103781 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.111666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.110933 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h"] Oct 30 06:38:02.125918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.125886 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cdae46b39755\"" Oct 30 06:38:02.138679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.138645 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-8z4m5\"" Oct 30 06:38:02.138883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.138722 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 06:38:02.138883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.138741 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 06:38:02.138883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.138805 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 06:38:02.138883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.138818 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 06:38:02.212091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.212042 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5fs\" (UniqueName: \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-kube-api-access-4h5fs\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.212417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.212391 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-plugins-config-volume\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.212527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.212503 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-routing-overlay-volume\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.212608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.212593 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-provider-credentials\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.323582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.323539 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-plugins-config-volume\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.323877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.323857 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-routing-overlay-volume\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.324039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.324022 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-provider-credentials\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.324411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.324388 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5fs\" (UniqueName: \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-kube-api-access-4h5fs\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.326356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.326333 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-plugins-config-volume\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.327086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.327064 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-routing-overlay-volume\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.363382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.362550 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5fs\" (UniqueName: \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-kube-api-access-4h5fs\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.386332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.375535 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-provider-credentials\") pod \"payload-processing-external-model-69f585c47c-wkp5h\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.445327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.444649 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:02.974341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:02.972711 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h"] Oct 30 06:38:03.749560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:03.749516 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" event={"ID":"9aaf92ff-2a28-4d87-97e9-06e555adb83d","Type":"ContainerStarted","Data":"d07b68051f807b268e9c61ef795c45d59cff3f5be51d2583aa46a7c00a9cf09b"} Oct 30 06:38:03.750133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:03.749581 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" event={"ID":"9aaf92ff-2a28-4d87-97e9-06e555adb83d","Type":"ContainerStarted","Data":"ff9f6f2d12a359c706ca9d14c36b97a94d5ca44e38b7e1b82a3d7d2d556f6bb2"} Oct 30 06:38:03.750133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:03.749609 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:03.792214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:03.792155 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" podStartSLOduration=1.7921394240000001 podStartE2EDuration="1.792139424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:03.792098719 +0000 UTC m=+2697.788055729" watchObservedRunningTime="2026-10-30 06:38:03.792139424 +0000 UTC m=+2697.788096436" Oct 30 06:38:05.038109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:05.038068 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt"] Oct 30 06:38:05.050987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:05.050955 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d78318cf-hvzbt"] Oct 30 06:38:06.121173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:06.121139 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac6147c-2278-4bc2-9a3a-12fecfdd4ae0" path="/var/lib/kubelet/pods/dac6147c-2278-4bc2-9a3a-12fecfdd4ae0/volumes" Oct 30 06:38:06.912779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:06.912751 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:38:06.932096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:06.932063 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:38:07.513493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.513458 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm"] Oct 30 06:38:07.519043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.519021 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:07.543021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.542997 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c44730-openshift-default-dockercfg-jsx6b\"" Oct 30 06:38:07.599447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.594220 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm"] Oct 30 06:38:07.663285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.663241 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istiod-ca-cert\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:07.665066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.663563 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-socket\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:07.665066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.663690 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-token\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:07.665066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.663772 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-credential-socket\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:07.665066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.663822 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-podinfo\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:07.665066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.663861 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-data\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:07.665066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.663907 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29km\" (UniqueName: \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-kube-api-access-g29km\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:07.665066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.664063 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-certs\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:07.665066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.664120 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-envoy\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:07.809393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.776735 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-certs\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.032508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:07.880848 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-certs\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.033031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.032989 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-envoy\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.033977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.033902 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istiod-ca-cert\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.034924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.034906 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-socket\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.038580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.035481 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-token\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.038888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.034840 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istiod-ca-cert\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.039081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.035297 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-socket\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.039286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.038873 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-credential-socket\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.045577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.039612 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-podinfo\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.045577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.045548 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-data\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.045814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.045619 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g29km\" (UniqueName: \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-kube-api-access-g29km\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.046521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.046479 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-data\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.046655 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.039137 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-credential-socket\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.095388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.088030 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-podinfo\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.128332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.120547 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29km\" (UniqueName: \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-kube-api-access-g29km\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.138329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.137042 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-envoy\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.149333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.143236 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-token\") pod \"e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.163415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.163374 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:08.323958 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:38:08.320499 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:38:08.600402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.593686 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm"] Oct 30 06:38:08.608575 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:38:08.608023 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a2a8f5_6f9d_4b20_ac91_eb3049e582eb.slice/crio-450337f5474b7b788ec46ef135fdcb5bf097b570bc15b4332e89227e20636e6d WatchSource:0}: Error finding container 450337f5474b7b788ec46ef135fdcb5bf097b570bc15b4332e89227e20636e6d: Status 404 returned error can't find the container with id 450337f5474b7b788ec46ef135fdcb5bf097b570bc15b4332e89227e20636e6d Oct 30 06:38:08.611467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.611417 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:08.611558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.611503 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:08.611558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:08.611540 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:09.004915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:09.004810 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" event={"ID":"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb","Type":"ContainerStarted","Data":"cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672"} Oct 30 06:38:09.004915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:09.004879 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" event={"ID":"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb","Type":"ContainerStarted","Data":"450337f5474b7b788ec46ef135fdcb5bf097b570bc15b4332e89227e20636e6d"} Oct 30 06:38:09.091342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:09.088540 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" podStartSLOduration=2.088517716 podStartE2EDuration="2.088517716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:09.067944577 +0000 UTC m=+2703.063901592" watchObservedRunningTime="2026-10-30 06:38:09.088517716 +0000 UTC m=+2703.084474726" Oct 30 06:38:09.175332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:09.174646 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:09.184590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:09.180423 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 30 06:38:09.184590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:09.180494 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 30 06:38:10.193501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:10.193462 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:10.392331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:10.392035 5001 scope.go:122] "RemoveContainer" containerID="b34d56118d91d82c526fd4b50d9dabbcaee9cc1c3ac7002cc2e013eef44faa9f" Oct 30 06:38:10.428745 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:10.428699 5001 scope.go:122] "RemoveContainer" containerID="729170e252b2d1cf77f627e30df79a6e35dce4a07573e1c0ae8ceb657f5d78e5" Oct 30 06:38:11.031286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:11.031258 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:11.032694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:11.032670 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:38:14.798912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:14.798878 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:38:16.138286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:16.138252 5001 generic.go:356] "Generic (PLEG): container finished" podID="981ae7ff-0c71-468e-b045-9f7735860591" containerID="0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc" exitCode=0 Oct 30 06:38:16.138793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:16.138328 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" event={"ID":"981ae7ff-0c71-468e-b045-9f7735860591","Type":"ContainerDied","Data":"0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc"} Oct 30 06:38:16.138885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:16.138862 5001 scope.go:122] "RemoveContainer" containerID="0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc" Oct 30 06:38:16.140926 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:16.140890 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:16.141066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:16.140979 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:16.141066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:16.141027 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:16.464536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:16.464277 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:38:17.161341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:17.161010 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" event={"ID":"981ae7ff-0c71-468e-b045-9f7735860591","Type":"ContainerStarted","Data":"4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c"} Oct 30 06:38:17.688334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:17.687715 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:38:17.688334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:17.687853 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 06:38:17.688334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:17.687910 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 06:38:18.701051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:18.701009 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 06:38:18.701669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:18.701072 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 06:38:19.685867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:19.685826 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 06:38:19.686190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:19.686161 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 06:38:20.681931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:20.681894 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 06:38:20.682626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:20.682576 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 06:38:21.088595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.088502 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7"] Oct 30 06:38:21.093910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.093886 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:21.108841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.108816 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7"] Oct 30 06:38:21.114830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.114793 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:38:21.114990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.114922 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:38:21.114990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.114986 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:38:21.122236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.122209 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:38:21.207527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.207494 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42c9c6ee-g4df7\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:21.207748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.207664 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42c9c6ee-g4df7\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:21.207831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.207813 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9j6p\" (UniqueName: \"kubernetes.io/projected/8c743e3a-4fd3-4320-8efa-b977cadd7338-kube-api-access-m9j6p\") pod \"maas-api-revoke-sub-42c9c6ee-g4df7\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:21.309587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.309554 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42c9c6ee-g4df7\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:21.309814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.309636 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9j6p\" (UniqueName: \"kubernetes.io/projected/8c743e3a-4fd3-4320-8efa-b977cadd7338-kube-api-access-m9j6p\") pod \"maas-api-revoke-sub-42c9c6ee-g4df7\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:21.309814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.309802 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42c9c6ee-g4df7\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:21.310433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.310410 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42c9c6ee-g4df7\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:21.312248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.312225 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42c9c6ee-g4df7\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:21.333967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.333943 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9j6p\" (UniqueName: \"kubernetes.io/projected/8c743e3a-4fd3-4320-8efa-b977cadd7338-kube-api-access-m9j6p\") pod \"maas-api-revoke-sub-42c9c6ee-g4df7\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:21.415724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.415684 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:21.694394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.693772 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 06:38:21.694394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.693837 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 06:38:21.805983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:21.805869 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7"] Oct 30 06:38:21.821815 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:38:21.821770 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c743e3a_4fd3_4320_8efa_b977cadd7338.slice/crio-e224c7d3219ba42d98b44adbe7e6b71c3a151b90d58d3ed07f474362dd755c45 WatchSource:0}: Error finding container e224c7d3219ba42d98b44adbe7e6b71c3a151b90d58d3ed07f474362dd755c45: Status 404 returned error can't find the container with id e224c7d3219ba42d98b44adbe7e6b71c3a151b90d58d3ed07f474362dd755c45 Oct 30 06:38:22.028986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:22.028898 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:38:22.227346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:22.227247 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" event={"ID":"8c743e3a-4fd3-4320-8efa-b977cadd7338","Type":"ContainerStarted","Data":"e224c7d3219ba42d98b44adbe7e6b71c3a151b90d58d3ed07f474362dd755c45"} Oct 30 06:38:22.692139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:22.692039 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 30 06:38:22.692729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:22.692493 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 30 06:38:23.238098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:23.238059 5001 generic.go:356] "Generic (PLEG): container finished" podID="8c743e3a-4fd3-4320-8efa-b977cadd7338" containerID="8eef61748104fb72621f9ad0022fa43d3817730c409bc406e22c46b70e393e69" exitCode=0 Oct 30 06:38:23.238548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:23.238178 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" event={"ID":"8c743e3a-4fd3-4320-8efa-b977cadd7338","Type":"ContainerDied","Data":"8eef61748104fb72621f9ad0022fa43d3817730c409bc406e22c46b70e393e69"} Oct 30 06:38:23.685155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:23.685121 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:38:23.686608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:23.686582 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:38:24.442088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:24.442060 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:24.576397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:24.576266 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-openshift-ca\") pod \"8c743e3a-4fd3-4320-8efa-b977cadd7338\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " Oct 30 06:38:24.576583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:24.576405 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-xks-ca\") pod \"8c743e3a-4fd3-4320-8efa-b977cadd7338\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " Oct 30 06:38:24.576728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:24.576702 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-openshift-ca" pod "8c743e3a-4fd3-4320-8efa-b977cadd7338" (UID: "8c743e3a-4fd3-4320-8efa-b977cadd7338"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:24.576826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:24.576711 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c743e3a-4fd3-4320-8efa-b977cadd7338-kube-api-access-m9j6p\" (UniqueName: \"kubernetes.io/projected/8c743e3a-4fd3-4320-8efa-b977cadd7338-kube-api-access-m9j6p\") pod \"8c743e3a-4fd3-4320-8efa-b977cadd7338\" (UID: \"8c743e3a-4fd3-4320-8efa-b977cadd7338\") " Oct 30 06:38:24.577596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:24.577568 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:24.578877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:24.578851 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c743e3a-4fd3-4320-8efa-b977cadd7338-kube-api-access-m9j6p" pod "8c743e3a-4fd3-4320-8efa-b977cadd7338" (UID: "8c743e3a-4fd3-4320-8efa-b977cadd7338"). InnerVolumeSpecName "kube-api-access-m9j6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:24.578998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:24.578907 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-xks-ca" pod "8c743e3a-4fd3-4320-8efa-b977cadd7338" (UID: "8c743e3a-4fd3-4320-8efa-b977cadd7338"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:38:24.679271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:24.679238 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c743e3a-4fd3-4320-8efa-b977cadd7338-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:24.679271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:24.679266 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9j6p\" (UniqueName: \"kubernetes.io/projected/8c743e3a-4fd3-4320-8efa-b977cadd7338-kube-api-access-m9j6p\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:25.259774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:25.259745 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" Oct 30 06:38:25.260100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:25.259779 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7" event={"ID":"8c743e3a-4fd3-4320-8efa-b977cadd7338","Type":"ContainerDied","Data":"e224c7d3219ba42d98b44adbe7e6b71c3a151b90d58d3ed07f474362dd755c45"} Oct 30 06:38:25.260100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:25.259833 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e224c7d3219ba42d98b44adbe7e6b71c3a151b90d58d3ed07f474362dd755c45" Oct 30 06:38:28.032041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.032006 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj"] Oct 30 06:38:28.032862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.032846 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8c743e3a-4fd3-4320-8efa-b977cadd7338" containerName="revoke-keys" Oct 30 06:38:28.032921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.032863 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c743e3a-4fd3-4320-8efa-b977cadd7338" containerName="revoke-keys" Oct 30 06:38:28.033063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.033051 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c743e3a-4fd3-4320-8efa-b977cadd7338" containerName="revoke-keys" Oct 30 06:38:28.036230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.036213 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:28.056518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.056483 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj"] Oct 30 06:38:28.063096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.063066 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:38:28.063297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.063094 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:38:28.063462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.063133 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:38:28.063462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.063201 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:38:28.144014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.143968 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047c2e57-js6hj\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:28.144200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.144034 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmrt\" (UniqueName: \"kubernetes.io/projected/277389bd-3a5e-49ee-8821-6b6f4768086f-kube-api-access-xrmrt\") pod \"maas-api-revoke-sub-047c2e57-js6hj\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:28.144257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.144217 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047c2e57-js6hj\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:28.245927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.245886 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047c2e57-js6hj\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:28.246155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.245964 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmrt\" (UniqueName: \"kubernetes.io/projected/277389bd-3a5e-49ee-8821-6b6f4768086f-kube-api-access-xrmrt\") pod \"maas-api-revoke-sub-047c2e57-js6hj\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:28.246239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.246147 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047c2e57-js6hj\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:28.246753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.246725 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047c2e57-js6hj\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:28.248489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.248468 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047c2e57-js6hj\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:28.270351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.270290 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmrt\" (UniqueName: \"kubernetes.io/projected/277389bd-3a5e-49ee-8821-6b6f4768086f-kube-api-access-xrmrt\") pod \"maas-api-revoke-sub-047c2e57-js6hj\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:28.351285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.351203 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:28.671998 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:38:28.671964 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277389bd_3a5e_49ee_8821_6b6f4768086f.slice/crio-9fb116c17b4d6097d744f687c83cafc11306502b8d63c905eca369f6ff2b541a WatchSource:0}: Error finding container 9fb116c17b4d6097d744f687c83cafc11306502b8d63c905eca369f6ff2b541a: Status 404 returned error can't find the container with id 9fb116c17b4d6097d744f687c83cafc11306502b8d63c905eca369f6ff2b541a Oct 30 06:38:28.674762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:28.674731 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj"] Oct 30 06:38:29.303670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:29.303632 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" event={"ID":"277389bd-3a5e-49ee-8821-6b6f4768086f","Type":"ContainerStarted","Data":"a22f1ee71ed2fabf13080142e77960dd15bc44a18104afaf70dd8474165264a4"} Oct 30 06:38:29.304116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:29.303700 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" event={"ID":"277389bd-3a5e-49ee-8821-6b6f4768086f","Type":"ContainerStarted","Data":"9fb116c17b4d6097d744f687c83cafc11306502b8d63c905eca369f6ff2b541a"} Oct 30 06:38:29.348721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:29.348624 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" podStartSLOduration=1.348609791 podStartE2EDuration="1.348609791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:29.344820591 +0000 UTC m=+2723.340777603" watchObservedRunningTime="2026-10-30 06:38:29.348609791 +0000 UTC m=+2723.344566801" Oct 30 06:38:30.313949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:30.313915 5001 generic.go:356] "Generic (PLEG): container finished" podID="277389bd-3a5e-49ee-8821-6b6f4768086f" containerID="a22f1ee71ed2fabf13080142e77960dd15bc44a18104afaf70dd8474165264a4" exitCode=0 Oct 30 06:38:30.314423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:30.314057 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" event={"ID":"277389bd-3a5e-49ee-8821-6b6f4768086f","Type":"ContainerDied","Data":"a22f1ee71ed2fabf13080142e77960dd15bc44a18104afaf70dd8474165264a4"} Oct 30 06:38:31.539290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:31.539245 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:31.629894 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:31.629846 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-openshift-ca\") pod \"277389bd-3a5e-49ee-8821-6b6f4768086f\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " Oct 30 06:38:31.630358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:31.630194 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277389bd-3a5e-49ee-8821-6b6f4768086f-kube-api-access-xrmrt\" (UniqueName: \"kubernetes.io/projected/277389bd-3a5e-49ee-8821-6b6f4768086f-kube-api-access-xrmrt\") pod \"277389bd-3a5e-49ee-8821-6b6f4768086f\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " Oct 30 06:38:31.630358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:31.630280 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-xks-ca\") pod \"277389bd-3a5e-49ee-8821-6b6f4768086f\" (UID: \"277389bd-3a5e-49ee-8821-6b6f4768086f\") " Oct 30 06:38:31.631265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:31.630723 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-openshift-ca" pod "277389bd-3a5e-49ee-8821-6b6f4768086f" (UID: "277389bd-3a5e-49ee-8821-6b6f4768086f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:31.631265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:31.631200 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:31.633156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:31.633129 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-xks-ca" pod "277389bd-3a5e-49ee-8821-6b6f4768086f" (UID: "277389bd-3a5e-49ee-8821-6b6f4768086f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:38:31.633574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:31.633547 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277389bd-3a5e-49ee-8821-6b6f4768086f-kube-api-access-xrmrt" pod "277389bd-3a5e-49ee-8821-6b6f4768086f" (UID: "277389bd-3a5e-49ee-8821-6b6f4768086f"). InnerVolumeSpecName "kube-api-access-xrmrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:31.732523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:31.732483 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrmrt\" (UniqueName: \"kubernetes.io/projected/277389bd-3a5e-49ee-8821-6b6f4768086f-kube-api-access-xrmrt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:31.732523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:31.732514 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/277389bd-3a5e-49ee-8821-6b6f4768086f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:32.367806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:32.367705 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" event={"ID":"277389bd-3a5e-49ee-8821-6b6f4768086f","Type":"ContainerDied","Data":"9fb116c17b4d6097d744f687c83cafc11306502b8d63c905eca369f6ff2b541a"} Oct 30 06:38:32.367806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:32.367774 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fb116c17b4d6097d744f687c83cafc11306502b8d63c905eca369f6ff2b541a" Oct 30 06:38:32.367806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:32.367810 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj" Oct 30 06:38:32.417462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:32.417363 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5"] Oct 30 06:38:36.019981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.019910 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj"] Oct 30 06:38:36.022504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.022477 5001 cpu_manager.go:414] "containerMap: removing container" podUID="277389bd-3a5e-49ee-8821-6b6f4768086f" containerName="revoke-keys" Oct 30 06:38:36.022677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.022664 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277389bd-3a5e-49ee-8821-6b6f4768086f" containerName="revoke-keys" Oct 30 06:38:36.023232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.023213 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="277389bd-3a5e-49ee-8821-6b6f4768086f" containerName="revoke-keys" Oct 30 06:38:36.037131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.037069 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.049233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.049203 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj"] Oct 30 06:38:36.066021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.065982 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"agc-cred-4b4eb8b1d03a\"" Oct 30 06:38:36.066190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.066026 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"payload-processing-external-model-e2e--7e273d00-dockercfg-t26z8\"" Oct 30 06:38:36.303386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.298587 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m4cs\" (UniqueName: \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-kube-api-access-5m4cs\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.303386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.298867 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.303386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.299166 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.303386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.299224 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.401273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.401225 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.401539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.401518 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.401621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.401576 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.401683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.401655 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m4cs\" (UniqueName: \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-kube-api-access-5m4cs\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.406339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.404499 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.412326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.409968 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.444338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.439077 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m4cs\" (UniqueName: \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-kube-api-access-5m4cs\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.453013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.450150 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:36.656852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:36.656813 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:37.186599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.184502 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj"] Oct 30 06:38:37.196331 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:38:37.193833 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad96cca_8fa9_4cb6_a620_2e4cdd500d33.slice/crio-3a6c85b8bef1e3aa023598a7a5d68c7078a80e942c369b3b5dcca99f22b4228e WatchSource:0}: Error finding container 3a6c85b8bef1e3aa023598a7a5d68c7078a80e942c369b3b5dcca99f22b4228e: Status 404 returned error can't find the container with id 3a6c85b8bef1e3aa023598a7a5d68c7078a80e942c369b3b5dcca99f22b4228e Oct 30 06:38:37.445227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.443237 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" podUID="6ee85c60-c718-4b79-a952-14f256ca8be9" containerName="payload-processing" containerID="cri-o://e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b" gracePeriod=25 Oct 30 06:38:37.472348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.472233 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" event={"ID":"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33","Type":"ContainerStarted","Data":"3a6c85b8bef1e3aa023598a7a5d68c7078a80e942c369b3b5dcca99f22b4228e"} Oct 30 06:38:37.752208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.752176 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:38:37.851379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.851295 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-routing-overlay-volume\") pod \"6ee85c60-c718-4b79-a952-14f256ca8be9\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " Oct 30 06:38:37.851776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.851751 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-plugins-config-volume\") pod \"6ee85c60-c718-4b79-a952-14f256ca8be9\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " Oct 30 06:38:37.851987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.851744 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-routing-overlay-volume" pod "6ee85c60-c718-4b79-a952-14f256ca8be9" (UID: "6ee85c60-c718-4b79-a952-14f256ca8be9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:37.852154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.852126 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-plugins-config-volume" pod "6ee85c60-c718-4b79-a952-14f256ca8be9" (UID: "6ee85c60-c718-4b79-a952-14f256ca8be9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:37.852279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.852202 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-kube-api-access-lrgzc\" (UniqueName: \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-kube-api-access-lrgzc\") pod \"6ee85c60-c718-4b79-a952-14f256ca8be9\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " Oct 30 06:38:37.852279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.852241 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-provider-credentials\") pod \"6ee85c60-c718-4b79-a952-14f256ca8be9\" (UID: \"6ee85c60-c718-4b79-a952-14f256ca8be9\") " Oct 30 06:38:37.853060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.853032 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:37.853163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.853064 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee85c60-c718-4b79-a952-14f256ca8be9-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:37.855407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.855383 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-kube-api-access-lrgzc" pod "6ee85c60-c718-4b79-a952-14f256ca8be9" (UID: "6ee85c60-c718-4b79-a952-14f256ca8be9"). InnerVolumeSpecName "kube-api-access-lrgzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:37.855758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.855734 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-provider-credentials" pod "6ee85c60-c718-4b79-a952-14f256ca8be9" (UID: "6ee85c60-c718-4b79-a952-14f256ca8be9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:37.954453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.954402 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrgzc\" (UniqueName: \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-kube-api-access-lrgzc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:37.954453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:37.954429 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ee85c60-c718-4b79-a952-14f256ca8be9-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:38.268381 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:38:38.268343 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:38:38.486032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.485926 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" event={"ID":"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33","Type":"ContainerStarted","Data":"11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2"} Oct 30 06:38:38.486221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.486047 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:38.488021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.487995 5001 generic.go:356] "Generic (PLEG): container finished" podID="6ee85c60-c718-4b79-a952-14f256ca8be9" containerID="e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b" exitCode=0 Oct 30 06:38:38.488175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.488043 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" Oct 30 06:38:38.488175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.488082 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" event={"ID":"6ee85c60-c718-4b79-a952-14f256ca8be9","Type":"ContainerDied","Data":"e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b"} Oct 30 06:38:38.488175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.488152 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5" event={"ID":"6ee85c60-c718-4b79-a952-14f256ca8be9","Type":"ContainerDied","Data":"1ba94b96be5ccb58bf08bddd2a2c09bcd5e7dfd89b5b9ab5c427ad006b9d8191"} Oct 30 06:38:38.488386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.488181 5001 scope.go:122] "RemoveContainer" containerID="e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b" Oct 30 06:38:38.499817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.499793 5001 scope.go:122] "RemoveContainer" containerID="e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b" Oct 30 06:38:38.500137 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:38:38.500116 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b\": container with ID starting with e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b not found: ID does not exist" containerID="e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b" Oct 30 06:38:38.500207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.500145 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b"} err="failed to get container status \"e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b\": rpc error: code = NotFound desc = could not find container \"e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b\": container with ID starting with e91486fbcedbad510eed116961ff5940561270ad35734aab0c85e173e348257b not found: ID does not exist" Oct 30 06:38:38.532122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.532067 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" podStartSLOduration=3.532053504 podStartE2EDuration="3.532053504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:38.53150459 +0000 UTC m=+2732.527461601" watchObservedRunningTime="2026-10-30 06:38:38.532053504 +0000 UTC m=+2732.528010517" Oct 30 06:38:38.575208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.575172 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5"] Oct 30 06:38:38.581955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:38.581922 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-6669hx5j5"] Oct 30 06:38:39.383245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:39.383208 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28"] Oct 30 06:38:40.218330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:40.215231 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee85c60-c718-4b79-a952-14f256ca8be9" path="/var/lib/kubelet/pods/6ee85c60-c718-4b79-a952-14f256ca8be9/volumes" Oct 30 06:38:43.378332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.361540 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw"] Oct 30 06:38:43.378332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.369678 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6ee85c60-c718-4b79-a952-14f256ca8be9" containerName="payload-processing" Oct 30 06:38:43.378332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.369707 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee85c60-c718-4b79-a952-14f256ca8be9" containerName="payload-processing" Oct 30 06:38:43.378332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.371137 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee85c60-c718-4b79-a952-14f256ca8be9" containerName="payload-processing" Oct 30 06:38:43.764998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.764915 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw"] Oct 30 06:38:43.765437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.765413 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:43.832651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.830590 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"payload-processing-external-model-e2e--d83828a5-dockercfg-cqlg5\"" Oct 30 06:38:43.832651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.830888 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"agc-cred-c32bb415fa0e\"" Oct 30 06:38:43.934193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.934157 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzm8t\" (UniqueName: \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-kube-api-access-mzm8t\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:43.934426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.934279 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:43.934426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.934322 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:43.934542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:43.934473 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:44.035698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.035609 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:44.035895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.035776 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzm8t\" (UniqueName: \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-kube-api-access-mzm8t\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:44.035895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.035868 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:44.036058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.035898 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:44.036734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.036706 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:44.036880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.036846 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:44.038592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.038560 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:44.064048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.064021 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzm8t\" (UniqueName: \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-kube-api-access-mzm8t\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:44.207465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.207431 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:44.410534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.407686 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" podUID="fdf1e732-da1b-4560-a14f-970c9d19a8ff" containerName="payload-processing" containerID="cri-o://315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27" gracePeriod=25 Oct 30 06:38:44.527057 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:38:44.527026 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f84c23_5737_466e_b7a7_7054286afeaa.slice/crio-508f5321abc550837e15963708cc6efb566102e8ca2e05551d09930be89dacde WatchSource:0}: Error finding container 508f5321abc550837e15963708cc6efb566102e8ca2e05551d09930be89dacde: Status 404 returned error can't find the container with id 508f5321abc550837e15963708cc6efb566102e8ca2e05551d09930be89dacde Oct 30 06:38:44.530521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.530491 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw"] Oct 30 06:38:44.703025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.702994 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:38:44.718455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.718423 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-routing-overlay-volume\") pod \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " Oct 30 06:38:44.718763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.718741 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-kube-api-access-v6pf8\" (UniqueName: \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-kube-api-access-v6pf8\") pod \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " Oct 30 06:38:44.718852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.718793 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-routing-overlay-volume" pod "fdf1e732-da1b-4560-a14f-970c9d19a8ff" (UID: "fdf1e732-da1b-4560-a14f-970c9d19a8ff"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:44.718899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.718861 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-provider-credentials\") pod \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " Oct 30 06:38:44.718938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.718914 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-plugins-config-volume\") pod \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\" (UID: \"fdf1e732-da1b-4560-a14f-970c9d19a8ff\") " Oct 30 06:38:44.719352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.719294 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-plugins-config-volume" pod "fdf1e732-da1b-4560-a14f-970c9d19a8ff" (UID: "fdf1e732-da1b-4560-a14f-970c9d19a8ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:44.719769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.719617 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:44.719769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.719635 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdf1e732-da1b-4560-a14f-970c9d19a8ff-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:44.721157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.721134 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-kube-api-access-v6pf8" pod "fdf1e732-da1b-4560-a14f-970c9d19a8ff" (UID: "fdf1e732-da1b-4560-a14f-970c9d19a8ff"). InnerVolumeSpecName "kube-api-access-v6pf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:44.721291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.721274 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-provider-credentials" pod "fdf1e732-da1b-4560-a14f-970c9d19a8ff" (UID: "fdf1e732-da1b-4560-a14f-970c9d19a8ff"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:44.821390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.821357 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6pf8\" (UniqueName: \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-kube-api-access-v6pf8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:44.821390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.821388 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdf1e732-da1b-4560-a14f-970c9d19a8ff-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:44.902130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.902096 5001 generic.go:356] "Generic (PLEG): container finished" podID="fdf1e732-da1b-4560-a14f-970c9d19a8ff" containerID="315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27" exitCode=0 Oct 30 06:38:44.902358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.902147 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" Oct 30 06:38:44.902358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.902174 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" event={"ID":"fdf1e732-da1b-4560-a14f-970c9d19a8ff","Type":"ContainerDied","Data":"315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27"} Oct 30 06:38:44.902358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.902234 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28" event={"ID":"fdf1e732-da1b-4560-a14f-970c9d19a8ff","Type":"ContainerDied","Data":"22eda8d9d0c9b992d1be5a5a95e56e0e336dd4c8c68aa63a03b368278a7598f3"} Oct 30 06:38:44.902358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.902258 5001 scope.go:122] "RemoveContainer" containerID="315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27" Oct 30 06:38:44.904585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.904547 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" event={"ID":"66f84c23-5737-466e-b7a7-7054286afeaa","Type":"ContainerStarted","Data":"f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e"} Oct 30 06:38:44.904720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.904616 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" event={"ID":"66f84c23-5737-466e-b7a7-7054286afeaa","Type":"ContainerStarted","Data":"508f5321abc550837e15963708cc6efb566102e8ca2e05551d09930be89dacde"} Oct 30 06:38:44.904720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.904654 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:44.915158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.915135 5001 scope.go:122] "RemoveContainer" containerID="315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27" Oct 30 06:38:44.915583 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:38:44.915552 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27\": container with ID starting with 315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27 not found: ID does not exist" containerID="315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27" Oct 30 06:38:44.915682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.915596 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27"} err="failed to get container status \"315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27\": rpc error: code = NotFound desc = could not find container \"315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27\": container with ID starting with 315900819a899de04b6d328205c7298705363446eaed0b677abd55cbc4d19b27 not found: ID does not exist" Oct 30 06:38:44.960962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.960875 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28"] Oct 30 06:38:44.966071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:44.966042 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-84d8tlv28"] Oct 30 06:38:45.005713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:45.005660 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" podStartSLOduration=2.005644079 podStartE2EDuration="2.005644079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:45.003019159 +0000 UTC m=+2738.998976168" watchObservedRunningTime="2026-10-30 06:38:45.005644079 +0000 UTC m=+2739.001601089" Oct 30 06:38:46.063193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.063152 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s"] Oct 30 06:38:46.064145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.064123 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fdf1e732-da1b-4560-a14f-970c9d19a8ff" containerName="payload-processing" Oct 30 06:38:46.064145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.064144 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf1e732-da1b-4560-a14f-970c9d19a8ff" containerName="payload-processing" Oct 30 06:38:46.064347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.064335 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf1e732-da1b-4560-a14f-970c9d19a8ff" containerName="payload-processing" Oct 30 06:38:46.069686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.069663 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.079109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.079017 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz"] Oct 30 06:38:46.084202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.084169 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.102613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.102589 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hc86w\"" Oct 30 06:38:46.109392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.109286 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s"] Oct 30 06:38:46.132191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.132156 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf1e732-da1b-4560-a14f-970c9d19a8ff" path="/var/lib/kubelet/pods/fdf1e732-da1b-4560-a14f-970c9d19a8ff/volumes" Oct 30 06:38:46.132559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.132542 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz"] Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.149269 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.149384 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.149448 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.149496 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.149550 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.149594 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.149665 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.149724 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spb28\" (UniqueName: \"kubernetes.io/projected/aed4e7ce-ff6a-4e62-92da-33fc1092c705-kube-api-access-spb28\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.149789 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.149836 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff80c97c-2e4d-402f-88a3-4168cc366e05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.149923 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.150053 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.150116 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.150152 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.150228 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.150294 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpxvn\" (UniqueName: \"kubernetes.io/projected/ff80c97c-2e4d-402f-88a3-4168cc366e05-kube-api-access-kpxvn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.150354 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.152326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.150426 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.256334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.252515 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.274347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.274285 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.274621 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.274744 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.274823 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxvn\" (UniqueName: \"kubernetes.io/projected/ff80c97c-2e4d-402f-88a3-4168cc366e05-kube-api-access-kpxvn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.274865 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.274929 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.274981 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.275089 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.275150 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.275185 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.275268 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.275319 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.275391 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.275442 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spb28\" (UniqueName: \"kubernetes.io/projected/aed4e7ce-ff6a-4e62-92da-33fc1092c705-kube-api-access-spb28\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.275502 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.275549 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff80c97c-2e4d-402f-88a3-4168cc366e05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.275636 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.282952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.276250 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.286857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.286170 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.286857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.286475 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.287086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.287057 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.287400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.287373 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.287628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.287610 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.287848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.287828 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.288222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.288199 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.289222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.289183 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff80c97c-2e4d-402f-88a3-4168cc366e05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.289396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.289218 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.289494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.289452 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.291796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.291774 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.292085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.292063 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.292167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.292099 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.310407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.310263 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff80c97c-2e4d-402f-88a3-4168cc366e05-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.310752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.310714 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed4e7ce-ff6a-4e62-92da-33fc1092c705-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.313093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.313065 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpxvn\" (UniqueName: \"kubernetes.io/projected/ff80c97c-2e4d-402f-88a3-4168cc366e05-kube-api-access-kpxvn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6zsz\" (UID: \"ff80c97c-2e4d-402f-88a3-4168cc366e05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.314258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.314196 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spb28\" (UniqueName: \"kubernetes.io/projected/aed4e7ce-ff6a-4e62-92da-33fc1092c705-kube-api-access-spb28\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5nt9s\" (UID: \"aed4e7ce-ff6a-4e62-92da-33fc1092c705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.387826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.387785 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:46.405733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.405702 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:46.745794 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:38:46.745766 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed4e7ce_ff6a_4e62_92da_33fc1092c705.slice/crio-66977846b50d66f127f6939f37cd3033366030a8d1085ca6d58999526a719589 WatchSource:0}: Error finding container 66977846b50d66f127f6939f37cd3033366030a8d1085ca6d58999526a719589: Status 404 returned error can't find the container with id 66977846b50d66f127f6939f37cd3033366030a8d1085ca6d58999526a719589 Oct 30 06:38:46.745925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.745789 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s"] Oct 30 06:38:46.748956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.748914 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:46.749088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.749002 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:46.749088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.749049 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:46.752183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.752159 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz"] Oct 30 06:38:46.765993 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:38:46.765968 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff80c97c_2e4d_402f_88a3_4168cc366e05.slice/crio-acb3053b3fdf93ba83104582210b043630b880213b9b7705f910e4fc934168d9 WatchSource:0}: Error finding container acb3053b3fdf93ba83104582210b043630b880213b9b7705f910e4fc934168d9: Status 404 returned error can't find the container with id acb3053b3fdf93ba83104582210b043630b880213b9b7705f910e4fc934168d9 Oct 30 06:38:46.769091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.769052 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:46.769190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.769142 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:46.769190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.769175 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:38:46.928353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.928297 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" event={"ID":"aed4e7ce-ff6a-4e62-92da-33fc1092c705","Type":"ContainerStarted","Data":"e41339114e1e7b4afdf0f2855fcda04fc93b7801f22687e03b5cd921d0b45a4f"} Oct 30 06:38:46.928532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.928387 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" event={"ID":"aed4e7ce-ff6a-4e62-92da-33fc1092c705","Type":"ContainerStarted","Data":"66977846b50d66f127f6939f37cd3033366030a8d1085ca6d58999526a719589"} Oct 30 06:38:46.931404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.930650 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" event={"ID":"ff80c97c-2e4d-402f-88a3-4168cc366e05","Type":"ContainerStarted","Data":"daee150a63c8c1185f8566f61b6bc79f9065354c50b43f1158936285ae472a12"} Oct 30 06:38:46.931404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.930698 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" event={"ID":"ff80c97c-2e4d-402f-88a3-4168cc366e05","Type":"ContainerStarted","Data":"acb3053b3fdf93ba83104582210b043630b880213b9b7705f910e4fc934168d9"} Oct 30 06:38:46.982909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:46.982113 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" podStartSLOduration=0.982093011 podStartE2EDuration="982.093011ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:46.975877854 +0000 UTC m=+2740.971834877" watchObservedRunningTime="2026-10-30 06:38:46.982093011 +0000 UTC m=+2740.978050023" Oct 30 06:38:47.031538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:47.031415 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" podStartSLOduration=1.031392524 podStartE2EDuration="1.031392524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:47.022907077 +0000 UTC m=+2741.018864090" watchObservedRunningTime="2026-10-30 06:38:47.031392524 +0000 UTC m=+2741.027349535" Oct 30 06:38:47.388547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:47.388510 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:47.390071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:47.390041 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5nt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Oct 30 06:38:47.390199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:47.390094 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" podUID="aed4e7ce-ff6a-4e62-92da-33fc1092c705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Oct 30 06:38:47.406392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:47.406362 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:47.407858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:47.407833 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6zsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 30 06:38:47.407984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:47.407878 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" podUID="ff80c97c-2e4d-402f-88a3-4168cc366e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 30 06:38:48.390820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:48.390780 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5nt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Oct 30 06:38:48.391450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:48.391414 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" podUID="aed4e7ce-ff6a-4e62-92da-33fc1092c705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Oct 30 06:38:48.406810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:48.406773 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6zsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 30 06:38:48.407134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:48.407107 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" podUID="ff80c97c-2e4d-402f-88a3-4168cc366e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 30 06:38:49.392049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:49.392011 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5nt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Oct 30 06:38:49.392692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:49.392658 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" podUID="aed4e7ce-ff6a-4e62-92da-33fc1092c705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Oct 30 06:38:49.406736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:49.406701 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6zsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 30 06:38:49.406998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:49.406972 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" podUID="ff80c97c-2e4d-402f-88a3-4168cc366e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 30 06:38:49.520124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:49.518536 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:50.345066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:50.345027 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj"] Oct 30 06:38:50.390331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:50.389762 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5nt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Oct 30 06:38:50.390331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:50.389862 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" podUID="aed4e7ce-ff6a-4e62-92da-33fc1092c705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Oct 30 06:38:50.420516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:50.420473 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6zsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 30 06:38:50.421044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:50.420539 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" podUID="ff80c97c-2e4d-402f-88a3-4168cc366e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 30 06:38:51.394864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:51.394816 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5nt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Oct 30 06:38:51.395106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:51.394900 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" podUID="aed4e7ce-ff6a-4e62-92da-33fc1092c705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Oct 30 06:38:51.410353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:51.409851 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6zsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 30 06:38:51.410353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:51.409907 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" podUID="ff80c97c-2e4d-402f-88a3-4168cc366e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 30 06:38:52.390333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:52.389772 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5nt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Oct 30 06:38:52.390333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:52.389875 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" podUID="aed4e7ce-ff6a-4e62-92da-33fc1092c705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Oct 30 06:38:52.407346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:52.406759 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6zsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 30 06:38:52.407346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:52.406821 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" podUID="ff80c97c-2e4d-402f-88a3-4168cc366e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 30 06:38:53.389334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:53.388692 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5nt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Oct 30 06:38:53.389334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:53.388756 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" podUID="aed4e7ce-ff6a-4e62-92da-33fc1092c705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Oct 30 06:38:53.415251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:53.415214 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6zsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 30 06:38:53.415878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:53.415847 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" podUID="ff80c97c-2e4d-402f-88a3-4168cc366e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 30 06:38:54.388505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:54.388466 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5nt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Oct 30 06:38:54.388707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:54.388535 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" podUID="aed4e7ce-ff6a-4e62-92da-33fc1092c705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Oct 30 06:38:54.406279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:54.406248 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6zsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 30 06:38:54.406451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:54.406299 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" podUID="ff80c97c-2e4d-402f-88a3-4168cc366e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 30 06:38:55.346466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:55.346424 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" podUID="0ad96cca-8fa9-4cb6-a620-2e4cdd500d33" containerName="payload-processing" containerID="cri-o://11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2" gracePeriod=25 Oct 30 06:38:55.805641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:55.805610 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:55.929098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:55.927553 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:38:55.955596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:55.955562 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-provider-credentials\") pod \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " Oct 30 06:38:55.956216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:55.956189 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-plugins-config-volume\") pod \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " Oct 30 06:38:55.959484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:55.959459 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-routing-overlay-volume\") pod \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " Oct 30 06:38:55.959887 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:55.959854 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-kube-api-access-5m4cs\" (UniqueName: \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-kube-api-access-5m4cs\") pod \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\" (UID: \"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33\") " Oct 30 06:38:55.993826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:55.993741 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-plugins-config-volume" pod "0ad96cca-8fa9-4cb6-a620-2e4cdd500d33" (UID: "0ad96cca-8fa9-4cb6-a620-2e4cdd500d33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:55.994478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:55.994450 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-routing-overlay-volume" pod "0ad96cca-8fa9-4cb6-a620-2e4cdd500d33" (UID: "0ad96cca-8fa9-4cb6-a620-2e4cdd500d33"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:56.030833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.017555 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-provider-credentials" pod "0ad96cca-8fa9-4cb6-a620-2e4cdd500d33" (UID: "0ad96cca-8fa9-4cb6-a620-2e4cdd500d33"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:56.030833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.030695 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-kube-api-access-5m4cs" pod "0ad96cca-8fa9-4cb6-a620-2e4cdd500d33" (UID: "0ad96cca-8fa9-4cb6-a620-2e4cdd500d33"). InnerVolumeSpecName "kube-api-access-5m4cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:56.121197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.121157 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:56.121197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.121198 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:56.121558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.121213 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m4cs\" (UniqueName: \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-kube-api-access-5m4cs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:56.121558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.121227 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:38:56.402336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.400882 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" Oct 30 06:38:56.444157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.439391 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5nt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:38:56.444157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.439446 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" podUID="aed4e7ce-ff6a-4e62-92da-33fc1092c705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:38:56.452623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.452585 5001 generic.go:356] "Generic (PLEG): container finished" podID="0ad96cca-8fa9-4cb6-a620-2e4cdd500d33" containerID="11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2" exitCode=0 Oct 30 06:38:56.455015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.452850 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" event={"ID":"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33","Type":"ContainerDied","Data":"11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2"} Oct 30 06:38:56.459012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.458955 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj" event={"ID":"0ad96cca-8fa9-4cb6-a620-2e4cdd500d33","Type":"ContainerDied","Data":"3a6c85b8bef1e3aa023598a7a5d68c7078a80e942c369b3b5dcca99f22b4228e"} Oct 30 06:38:56.459239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.459199 5001 scope.go:122] "RemoveContainer" containerID="11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2" Oct 30 06:38:56.491248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.490198 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6zsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:38:56.491248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.490285 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" podUID="ff80c97c-2e4d-402f-88a3-4168cc366e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:38:56.549894 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.549755 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj"] Oct 30 06:38:56.571279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.570873 5001 scope.go:122] "RemoveContainer" containerID="11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2" Oct 30 06:38:56.572682 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:38:56.572652 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2\": container with ID starting with 11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2 not found: ID does not exist" containerID="11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2" Oct 30 06:38:56.572880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.572849 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2"} err="failed to get container status \"11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2\": rpc error: code = NotFound desc = could not find container \"11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2\": container with ID starting with 11d80fc4eea4a9f660461e4b7c5e8c512c2d7bbfda01079a80bc2f17a34c72d2 not found: ID does not exist" Oct 30 06:38:56.593335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.592417 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg"] Oct 30 06:38:56.593335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.592719 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" containerName="istio-proxy" containerID="cri-o://579a94f50bcce8ca6cc4dbd77073d4b07a4d1f3712a9055509e0d364a3710660" gracePeriod=30 Oct 30 06:38:56.757381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.756775 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47vdbxj"] Oct 30 06:38:56.822344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:56.798375 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw"] Oct 30 06:38:57.118976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:57.116343 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:57.119408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:57.119383 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:57.119540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:57.119477 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5nt9s" Oct 30 06:38:57.140212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:57.140174 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:57.141374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:57.140652 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:57.142680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:57.142661 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6zsz" Oct 30 06:38:58.217342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:38:58.217261 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad96cca-8fa9-4cb6-a620-2e4cdd500d33" path="/var/lib/kubelet/pods/0ad96cca-8fa9-4cb6-a620-2e4cdd500d33/volumes" Oct 30 06:39:01.813518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:01.803431 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" podUID="66f84c23-5737-466e-b7a7-7054286afeaa" containerName="payload-processing" containerID="cri-o://f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e" gracePeriod=25 Oct 30 06:39:01.887901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:01.885813 5001 generic.go:356] "Generic (PLEG): container finished" podID="33345dbd-38c7-4372-829a-607ef801ba0b" containerID="579a94f50bcce8ca6cc4dbd77073d4b07a4d1f3712a9055509e0d364a3710660" exitCode=0 Oct 30 06:39:01.887901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:01.885889 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" event={"ID":"33345dbd-38c7-4372-829a-607ef801ba0b","Type":"ContainerDied","Data":"579a94f50bcce8ca6cc4dbd77073d4b07a4d1f3712a9055509e0d364a3710660"} Oct 30 06:39:02.150661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.150631 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:39:02.156606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.156573 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-kube-api-access-rb9cx\" (UniqueName: \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-kube-api-access-rb9cx\") pod \"33345dbd-38c7-4372-829a-607ef801ba0b\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " Oct 30 06:39:02.156735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.156648 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33345dbd-38c7-4372-829a-607ef801ba0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33345dbd-38c7-4372-829a-607ef801ba0b-istiod-ca-cert\") pod \"33345dbd-38c7-4372-829a-607ef801ba0b\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " Oct 30 06:39:02.156807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.156786 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-envoy\") pod \"33345dbd-38c7-4372-829a-607ef801ba0b\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " Oct 30 06:39:02.156878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.156853 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-credential-socket\") pod \"33345dbd-38c7-4372-829a-607ef801ba0b\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " Oct 30 06:39:02.156983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.156963 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-data\") pod \"33345dbd-38c7-4372-829a-607ef801ba0b\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " Oct 30 06:39:02.157065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.157022 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-certs\") pod \"33345dbd-38c7-4372-829a-607ef801ba0b\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " Oct 30 06:39:02.157127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.157072 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-istio-token\" (UniqueName: \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-istio-token\") pod \"33345dbd-38c7-4372-829a-607ef801ba0b\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " Oct 30 06:39:02.157184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.157172 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-socket\") pod \"33345dbd-38c7-4372-829a-607ef801ba0b\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " Oct 30 06:39:02.157240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.157212 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33345dbd-38c7-4372-829a-607ef801ba0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33345dbd-38c7-4372-829a-607ef801ba0b-istio-podinfo\") pod \"33345dbd-38c7-4372-829a-607ef801ba0b\" (UID: \"33345dbd-38c7-4372-829a-607ef801ba0b\") " Oct 30 06:39:02.158450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.157429 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33345dbd-38c7-4372-829a-607ef801ba0b-istiod-ca-cert" pod "33345dbd-38c7-4372-829a-607ef801ba0b" (UID: "33345dbd-38c7-4372-829a-607ef801ba0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:02.158450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.157768 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-credential-socket" pod "33345dbd-38c7-4372-829a-607ef801ba0b" (UID: "33345dbd-38c7-4372-829a-607ef801ba0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:02.158450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.157704 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-socket" pod "33345dbd-38c7-4372-829a-607ef801ba0b" (UID: "33345dbd-38c7-4372-829a-607ef801ba0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:02.158450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.157795 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-data" pod "33345dbd-38c7-4372-829a-607ef801ba0b" (UID: "33345dbd-38c7-4372-829a-607ef801ba0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:02.158450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.157839 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-certs" pod "33345dbd-38c7-4372-829a-607ef801ba0b" (UID: "33345dbd-38c7-4372-829a-607ef801ba0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:02.158949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.158674 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.158949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.158696 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33345dbd-38c7-4372-829a-607ef801ba0b-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.158949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.158711 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.158949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.158725 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.158949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.158738 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.171516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.171474 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33345dbd-38c7-4372-829a-607ef801ba0b-istio-podinfo" pod "33345dbd-38c7-4372-829a-607ef801ba0b" (UID: "33345dbd-38c7-4372-829a-607ef801ba0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:39:02.174617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.174542 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-kube-api-access-rb9cx" pod "33345dbd-38c7-4372-829a-607ef801ba0b" (UID: "33345dbd-38c7-4372-829a-607ef801ba0b"). InnerVolumeSpecName "kube-api-access-rb9cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:02.174617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.174594 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-istio-token" pod "33345dbd-38c7-4372-829a-607ef801ba0b" (UID: "33345dbd-38c7-4372-829a-607ef801ba0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:02.174892 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.174672 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-envoy" pod "33345dbd-38c7-4372-829a-607ef801ba0b" (UID: "33345dbd-38c7-4372-829a-607ef801ba0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:02.263143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.263050 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33345dbd-38c7-4372-829a-607ef801ba0b-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.263143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.263086 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.263143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.263102 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33345dbd-38c7-4372-829a-607ef801ba0b-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.263143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.263116 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb9cx\" (UniqueName: \"kubernetes.io/projected/33345dbd-38c7-4372-829a-607ef801ba0b-kube-api-access-rb9cx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.297044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.296881 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:39:02.368472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.363687 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-kube-api-access-mzm8t\" (UniqueName: \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-kube-api-access-mzm8t\") pod \"66f84c23-5737-466e-b7a7-7054286afeaa\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " Oct 30 06:39:02.368472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.363738 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-plugins-config-volume\") pod \"66f84c23-5737-466e-b7a7-7054286afeaa\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " Oct 30 06:39:02.368472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.363813 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-routing-overlay-volume\") pod \"66f84c23-5737-466e-b7a7-7054286afeaa\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " Oct 30 06:39:02.368472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.363857 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-provider-credentials\") pod \"66f84c23-5737-466e-b7a7-7054286afeaa\" (UID: \"66f84c23-5737-466e-b7a7-7054286afeaa\") " Oct 30 06:39:02.368472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.365466 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-routing-overlay-volume" pod "66f84c23-5737-466e-b7a7-7054286afeaa" (UID: "66f84c23-5737-466e-b7a7-7054286afeaa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:02.368472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.367663 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-provider-credentials" pod "66f84c23-5737-466e-b7a7-7054286afeaa" (UID: "66f84c23-5737-466e-b7a7-7054286afeaa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:02.368472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.368432 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-plugins-config-volume" pod "66f84c23-5737-466e-b7a7-7054286afeaa" (UID: "66f84c23-5737-466e-b7a7-7054286afeaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:02.370961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.370927 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-kube-api-access-mzm8t" pod "66f84c23-5737-466e-b7a7-7054286afeaa" (UID: "66f84c23-5737-466e-b7a7-7054286afeaa"). InnerVolumeSpecName "kube-api-access-mzm8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:02.472558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.472522 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.472558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.472553 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.472558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.472564 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzm8t\" (UniqueName: \"kubernetes.io/projected/66f84c23-5737-466e-b7a7-7054286afeaa-kube-api-access-mzm8t\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.472897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.472573 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66f84c23-5737-466e-b7a7-7054286afeaa-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:02.898083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.898047 5001 generic.go:356] "Generic (PLEG): container finished" podID="66f84c23-5737-466e-b7a7-7054286afeaa" containerID="f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e" exitCode=0 Oct 30 06:39:02.898590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.898102 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" Oct 30 06:39:02.898590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.898130 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" event={"ID":"66f84c23-5737-466e-b7a7-7054286afeaa","Type":"ContainerDied","Data":"f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e"} Oct 30 06:39:02.898590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.898184 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw" event={"ID":"66f84c23-5737-466e-b7a7-7054286afeaa","Type":"ContainerDied","Data":"508f5321abc550837e15963708cc6efb566102e8ca2e05551d09930be89dacde"} Oct 30 06:39:02.898590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.898208 5001 scope.go:122] "RemoveContainer" containerID="f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e" Oct 30 06:39:02.900268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.900225 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" event={"ID":"33345dbd-38c7-4372-829a-607ef801ba0b","Type":"ContainerDied","Data":"8f923fcf0090e898b9dc17fd74bad7696f77deb4cee80051cd0b4ee8ed9e3bf6"} Oct 30 06:39:02.900268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.900263 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg" Oct 30 06:39:02.927753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.927728 5001 scope.go:122] "RemoveContainer" containerID="f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e" Oct 30 06:39:02.928124 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:39:02.928098 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e\": container with ID starting with f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e not found: ID does not exist" containerID="f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e" Oct 30 06:39:02.928247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.928131 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e"} err="failed to get container status \"f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e\": rpc error: code = NotFound desc = could not find container \"f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e\": container with ID starting with f9d80ea2ad051c2b12402eceb7429e85ba728e76c4b12f648f0373c7460c945e not found: ID does not exist" Oct 30 06:39:02.928247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.928150 5001 scope.go:122] "RemoveContainer" containerID="579a94f50bcce8ca6cc4dbd77073d4b07a4d1f3712a9055509e0d364a3710660" Oct 30 06:39:02.956204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.956169 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw"] Oct 30 06:39:02.969475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:02.969438 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6ftdqpw"] Oct 30 06:39:03.022774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:03.022737 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg"] Oct 30 06:39:03.032588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:03.032556 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-249sg"] Oct 30 06:39:04.122859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:04.122820 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" path="/var/lib/kubelet/pods/33345dbd-38c7-4372-829a-607ef801ba0b/volumes" Oct 30 06:39:04.123510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:04.123486 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f84c23-5737-466e-b7a7-7054286afeaa" path="/var/lib/kubelet/pods/66f84c23-5737-466e-b7a7-7054286afeaa/volumes" Oct 30 06:39:06.198574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.198537 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp"] Oct 30 06:39:06.199884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.199857 5001 cpu_manager.go:414] "containerMap: removing container" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" containerName="istio-proxy" Oct 30 06:39:06.199884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.199882 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" containerName="istio-proxy" Oct 30 06:39:06.200115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.199901 5001 cpu_manager.go:414] "containerMap: removing container" podUID="66f84c23-5737-466e-b7a7-7054286afeaa" containerName="payload-processing" Oct 30 06:39:06.200115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.199909 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f84c23-5737-466e-b7a7-7054286afeaa" containerName="payload-processing" Oct 30 06:39:06.200115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.199978 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0ad96cca-8fa9-4cb6-a620-2e4cdd500d33" containerName="payload-processing" Oct 30 06:39:06.200115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.199986 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad96cca-8fa9-4cb6-a620-2e4cdd500d33" containerName="payload-processing" Oct 30 06:39:06.200372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.200202 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad96cca-8fa9-4cb6-a620-2e4cdd500d33" containerName="payload-processing" Oct 30 06:39:06.200372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.200225 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="33345dbd-38c7-4372-829a-607ef801ba0b" containerName="istio-proxy" Oct 30 06:39:06.200372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.200236 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f84c23-5737-466e-b7a7-7054286afeaa" containerName="payload-processing" Oct 30 06:39:06.204574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.204551 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:06.221319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.221274 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp"] Oct 30 06:39:06.231358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.226976 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:39:06.231358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.227078 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:39:06.231358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.227083 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:39:06.233226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.233204 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:39:06.259864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.259820 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlz45\" (UniqueName: \"kubernetes.io/projected/130ca010-84f5-4c85-a641-3e6456b30222-kube-api-access-rlz45\") pod \"maas-api-revoke-sub-2de17a6c-djcwp\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:06.260056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.260025 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de17a6c-djcwp\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:06.260152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.260126 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de17a6c-djcwp\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:06.362030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.361995 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de17a6c-djcwp\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:06.362223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.362129 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de17a6c-djcwp\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:06.362275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.362261 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlz45\" (UniqueName: \"kubernetes.io/projected/130ca010-84f5-4c85-a641-3e6456b30222-kube-api-access-rlz45\") pod \"maas-api-revoke-sub-2de17a6c-djcwp\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:06.362870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.362842 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2de17a6c-djcwp\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:06.364736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.364717 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2de17a6c-djcwp\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:06.387359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.387326 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlz45\" (UniqueName: \"kubernetes.io/projected/130ca010-84f5-4c85-a641-3e6456b30222-kube-api-access-rlz45\") pod \"maas-api-revoke-sub-2de17a6c-djcwp\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:06.523394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.523285 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:06.847753 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:39:06.847722 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130ca010_84f5_4c85_a641_3e6456b30222.slice/crio-26b5f33354caf2e1843ec4517aadde4fe9952f40407845d1b9c77bbc31a6660c WatchSource:0}: Error finding container 26b5f33354caf2e1843ec4517aadde4fe9952f40407845d1b9c77bbc31a6660c: Status 404 returned error can't find the container with id 26b5f33354caf2e1843ec4517aadde4fe9952f40407845d1b9c77bbc31a6660c Oct 30 06:39:06.848318 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.848274 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp"] Oct 30 06:39:06.950532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:06.950501 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" event={"ID":"130ca010-84f5-4c85-a641-3e6456b30222","Type":"ContainerStarted","Data":"26b5f33354caf2e1843ec4517aadde4fe9952f40407845d1b9c77bbc31a6660c"} Oct 30 06:39:07.964351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:07.964279 5001 generic.go:356] "Generic (PLEG): container finished" podID="130ca010-84f5-4c85-a641-3e6456b30222" containerID="0996e849b4a17c2aec0bd625d31a4010dcce6e910edf81475a0b51b3a9358daa" exitCode=0 Oct 30 06:39:07.964351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:07.964343 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" event={"ID":"130ca010-84f5-4c85-a641-3e6456b30222","Type":"ContainerDied","Data":"0996e849b4a17c2aec0bd625d31a4010dcce6e910edf81475a0b51b3a9358daa"} Oct 30 06:39:08.270894 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:39:08.270805 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:39:09.197827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.197793 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:09.348958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.348880 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130ca010-84f5-4c85-a641-3e6456b30222-kube-api-access-rlz45\" (UniqueName: \"kubernetes.io/projected/130ca010-84f5-4c85-a641-3e6456b30222-kube-api-access-rlz45\") pod \"130ca010-84f5-4c85-a641-3e6456b30222\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " Oct 30 06:39:09.349116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.349107 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-openshift-ca\") pod \"130ca010-84f5-4c85-a641-3e6456b30222\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " Oct 30 06:39:09.349194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.349180 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-xks-ca\") pod \"130ca010-84f5-4c85-a641-3e6456b30222\" (UID: \"130ca010-84f5-4c85-a641-3e6456b30222\") " Oct 30 06:39:09.349514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.349482 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-openshift-ca" pod "130ca010-84f5-4c85-a641-3e6456b30222" (UID: "130ca010-84f5-4c85-a641-3e6456b30222"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:09.349818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.349708 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:09.351273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.351250 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130ca010-84f5-4c85-a641-3e6456b30222-kube-api-access-rlz45" pod "130ca010-84f5-4c85-a641-3e6456b30222" (UID: "130ca010-84f5-4c85-a641-3e6456b30222"). InnerVolumeSpecName "kube-api-access-rlz45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:09.351398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.351347 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-xks-ca" pod "130ca010-84f5-4c85-a641-3e6456b30222" (UID: "130ca010-84f5-4c85-a641-3e6456b30222"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:39:09.451384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.451342 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlz45\" (UniqueName: \"kubernetes.io/projected/130ca010-84f5-4c85-a641-3e6456b30222-kube-api-access-rlz45\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:09.451384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.451371 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130ca010-84f5-4c85-a641-3e6456b30222-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:09.989537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.989503 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" Oct 30 06:39:09.989754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.989532 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp" event={"ID":"130ca010-84f5-4c85-a641-3e6456b30222","Type":"ContainerDied","Data":"26b5f33354caf2e1843ec4517aadde4fe9952f40407845d1b9c77bbc31a6660c"} Oct 30 06:39:09.989754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:09.989592 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26b5f33354caf2e1843ec4517aadde4fe9952f40407845d1b9c77bbc31a6660c" Oct 30 06:39:10.536962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:10.536927 5001 scope.go:122] "RemoveContainer" containerID="a0ef9d5c4283b1743eabc86e6f2e10e495319312aebdeae4f2c12b49f1491521" Oct 30 06:39:10.622330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:10.612884 5001 scope.go:122] "RemoveContainer" containerID="b6d93db729c4a873fd663373387a63b3d6ae97d0e4e4f795684029ce5c998289" Oct 30 06:39:10.656433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:10.651996 5001 scope.go:122] "RemoveContainer" containerID="9c82b9591544b4044141d85e0e20dc6ce4e6ca86fb3033f82fd707aa3641cc89" Oct 30 06:39:10.710136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:10.710105 5001 scope.go:122] "RemoveContainer" containerID="433a777e81af1df4bf0596728995511772638c325f421913fb961a2dcf823fa7" Oct 30 06:39:18.056355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.055487 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt"] Oct 30 06:39:18.071330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.056914 5001 cpu_manager.go:414] "containerMap: removing container" podUID="130ca010-84f5-4c85-a641-3e6456b30222" containerName="revoke-keys" Oct 30 06:39:18.071330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.056946 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130ca010-84f5-4c85-a641-3e6456b30222" containerName="revoke-keys" Oct 30 06:39:18.071330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.057213 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="130ca010-84f5-4c85-a641-3e6456b30222" containerName="revoke-keys" Oct 30 06:39:18.118346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.117448 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:18.142347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.137298 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:39:18.142347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.137414 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:39:18.154344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.152057 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:39:18.154344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.152175 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:39:18.239352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.208526 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zwg\" (UniqueName: \"kubernetes.io/projected/eb485d73-259f-44ca-be8f-c015ba4e9c80-kube-api-access-b5zwg\") pod \"maas-api-revoke-sub-695c9bf9-7v9zt\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:18.239352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.208929 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695c9bf9-7v9zt\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:18.239352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.209199 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695c9bf9-7v9zt\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:18.321248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.312288 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695c9bf9-7v9zt\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:18.323067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.323036 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695c9bf9-7v9zt\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:18.325035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.325004 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zwg\" (UniqueName: \"kubernetes.io/projected/eb485d73-259f-44ca-be8f-c015ba4e9c80-kube-api-access-b5zwg\") pod \"maas-api-revoke-sub-695c9bf9-7v9zt\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:18.345680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.322255 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695c9bf9-7v9zt\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:18.349863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.349837 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt"] Oct 30 06:39:18.379227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.379197 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695c9bf9-7v9zt\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:18.384547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.384518 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zwg\" (UniqueName: \"kubernetes.io/projected/eb485d73-259f-44ca-be8f-c015ba4e9c80-kube-api-access-b5zwg\") pod \"maas-api-revoke-sub-695c9bf9-7v9zt\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:18.513356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.513322 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:18.824995 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:39:18.824908 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb485d73_259f_44ca_be8f_c015ba4e9c80.slice/crio-47169eeb3385b76c8cbb571cb2f208db4bceb152d14eaec406ef10879757e3f8 WatchSource:0}: Error finding container 47169eeb3385b76c8cbb571cb2f208db4bceb152d14eaec406ef10879757e3f8: Status 404 returned error can't find the container with id 47169eeb3385b76c8cbb571cb2f208db4bceb152d14eaec406ef10879757e3f8 Oct 30 06:39:18.825798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:18.825772 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt"] Oct 30 06:39:19.355361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:19.355282 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" event={"ID":"eb485d73-259f-44ca-be8f-c015ba4e9c80","Type":"ContainerStarted","Data":"47169eeb3385b76c8cbb571cb2f208db4bceb152d14eaec406ef10879757e3f8"} Oct 30 06:39:20.366783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:20.366749 5001 generic.go:356] "Generic (PLEG): container finished" podID="eb485d73-259f-44ca-be8f-c015ba4e9c80" containerID="323db2101c1dea379da8f307108200f1400662e8eaeb1fff8f36062173980ec3" exitCode=0 Oct 30 06:39:20.367295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:20.366826 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" event={"ID":"eb485d73-259f-44ca-be8f-c015ba4e9c80","Type":"ContainerDied","Data":"323db2101c1dea379da8f307108200f1400662e8eaeb1fff8f36062173980ec3"} Oct 30 06:39:21.659086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:21.659016 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:21.731747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:21.731710 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb485d73-259f-44ca-be8f-c015ba4e9c80-kube-api-access-b5zwg\" (UniqueName: \"kubernetes.io/projected/eb485d73-259f-44ca-be8f-c015ba4e9c80-kube-api-access-b5zwg\") pod \"eb485d73-259f-44ca-be8f-c015ba4e9c80\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " Oct 30 06:39:21.731984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:21.731842 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-xks-ca\") pod \"eb485d73-259f-44ca-be8f-c015ba4e9c80\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " Oct 30 06:39:21.732218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:21.732195 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-openshift-ca\") pod \"eb485d73-259f-44ca-be8f-c015ba4e9c80\" (UID: \"eb485d73-259f-44ca-be8f-c015ba4e9c80\") " Oct 30 06:39:21.732607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:21.732577 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-openshift-ca" pod "eb485d73-259f-44ca-be8f-c015ba4e9c80" (UID: "eb485d73-259f-44ca-be8f-c015ba4e9c80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:21.733439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:21.733415 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:21.736832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:21.736794 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb485d73-259f-44ca-be8f-c015ba4e9c80-kube-api-access-b5zwg" pod "eb485d73-259f-44ca-be8f-c015ba4e9c80" (UID: "eb485d73-259f-44ca-be8f-c015ba4e9c80"). InnerVolumeSpecName "kube-api-access-b5zwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:21.739108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:21.739069 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-xks-ca" pod "eb485d73-259f-44ca-be8f-c015ba4e9c80" (UID: "eb485d73-259f-44ca-be8f-c015ba4e9c80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:39:21.835132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:21.835091 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5zwg\" (UniqueName: \"kubernetes.io/projected/eb485d73-259f-44ca-be8f-c015ba4e9c80-kube-api-access-b5zwg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:21.835132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:21.835134 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb485d73-259f-44ca-be8f-c015ba4e9c80-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:22.387352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:22.387238 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" Oct 30 06:39:22.387352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:22.387276 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt" event={"ID":"eb485d73-259f-44ca-be8f-c015ba4e9c80","Type":"ContainerDied","Data":"47169eeb3385b76c8cbb571cb2f208db4bceb152d14eaec406ef10879757e3f8"} Oct 30 06:39:22.387560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:22.387368 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47169eeb3385b76c8cbb571cb2f208db4bceb152d14eaec406ef10879757e3f8" Oct 30 06:39:30.047610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.047573 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j"] Oct 30 06:39:30.048512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.048493 5001 cpu_manager.go:414] "containerMap: removing container" podUID="eb485d73-259f-44ca-be8f-c015ba4e9c80" containerName="revoke-keys" Oct 30 06:39:30.048595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.048514 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb485d73-259f-44ca-be8f-c015ba4e9c80" containerName="revoke-keys" Oct 30 06:39:30.048693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.048684 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb485d73-259f-44ca-be8f-c015ba4e9c80" containerName="revoke-keys" Oct 30 06:39:30.066012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.065976 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:30.074271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.074242 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j"] Oct 30 06:39:30.086691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.086663 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:39:30.087052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.087029 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:39:30.099793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.099759 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:39:30.099951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.099759 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:39:30.222331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.220541 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpxg\" (UniqueName: \"kubernetes.io/projected/7a7ad224-962d-45ab-b0ca-96c5240a14bf-kube-api-access-qgpxg\") pod \"maas-api-revoke-sub-f578085a-zcx9j\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:30.222331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.220663 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f578085a-zcx9j\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:30.222331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.220859 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f578085a-zcx9j\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:30.326707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.326604 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f578085a-zcx9j\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:30.329676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.329647 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpxg\" (UniqueName: \"kubernetes.io/projected/7a7ad224-962d-45ab-b0ca-96c5240a14bf-kube-api-access-qgpxg\") pod \"maas-api-revoke-sub-f578085a-zcx9j\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:30.335561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.335527 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f578085a-zcx9j\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:30.335949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.329280 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f578085a-zcx9j\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:30.365185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.365152 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpxg\" (UniqueName: \"kubernetes.io/projected/7a7ad224-962d-45ab-b0ca-96c5240a14bf-kube-api-access-qgpxg\") pod \"maas-api-revoke-sub-f578085a-zcx9j\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:30.376746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.376719 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f578085a-zcx9j\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:30.390628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.390589 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:30.789343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:30.788760 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j"] Oct 30 06:39:30.791006 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:39:30.790978 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7ad224_962d_45ab_b0ca_96c5240a14bf.slice/crio-4c9600a021826053fbb1d3dccb1e6e8c0dd72d144c035fc08cce40262dcadc83 WatchSource:0}: Error finding container 4c9600a021826053fbb1d3dccb1e6e8c0dd72d144c035fc08cce40262dcadc83: Status 404 returned error can't find the container with id 4c9600a021826053fbb1d3dccb1e6e8c0dd72d144c035fc08cce40262dcadc83 Oct 30 06:39:31.483875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:31.483833 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" event={"ID":"7a7ad224-962d-45ab-b0ca-96c5240a14bf","Type":"ContainerStarted","Data":"d06cd38f8f1562292a9fffcb5ddca28ecb080ed9ca154c8352102b57ec0c4206"} Oct 30 06:39:31.484441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:31.483905 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" event={"ID":"7a7ad224-962d-45ab-b0ca-96c5240a14bf","Type":"ContainerStarted","Data":"4c9600a021826053fbb1d3dccb1e6e8c0dd72d144c035fc08cce40262dcadc83"} Oct 30 06:39:31.531333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:31.529189 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" podStartSLOduration=1.529169215 podStartE2EDuration="1.529169215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:39:31.525449963 +0000 UTC m=+2785.521406973" watchObservedRunningTime="2026-10-30 06:39:31.529169215 +0000 UTC m=+2785.525126225" Oct 30 06:39:31.867890 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:39:31.867848 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7ad224_962d_45ab_b0ca_96c5240a14bf.slice/crio-conmon-d06cd38f8f1562292a9fffcb5ddca28ecb080ed9ca154c8352102b57ec0c4206.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7ad224_962d_45ab_b0ca_96c5240a14bf.slice/crio-d06cd38f8f1562292a9fffcb5ddca28ecb080ed9ca154c8352102b57ec0c4206.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:39:32.496438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:32.496400 5001 generic.go:356] "Generic (PLEG): container finished" podID="7a7ad224-962d-45ab-b0ca-96c5240a14bf" containerID="d06cd38f8f1562292a9fffcb5ddca28ecb080ed9ca154c8352102b57ec0c4206" exitCode=0 Oct 30 06:39:32.496966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:32.496501 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" event={"ID":"7a7ad224-962d-45ab-b0ca-96c5240a14bf","Type":"ContainerDied","Data":"d06cd38f8f1562292a9fffcb5ddca28ecb080ed9ca154c8352102b57ec0c4206"} Oct 30 06:39:33.725841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:33.725815 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:33.907103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:33.907057 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-xks-ca\") pod \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " Oct 30 06:39:33.907386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:33.907249 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7ad224-962d-45ab-b0ca-96c5240a14bf-kube-api-access-qgpxg\" (UniqueName: \"kubernetes.io/projected/7a7ad224-962d-45ab-b0ca-96c5240a14bf-kube-api-access-qgpxg\") pod \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " Oct 30 06:39:33.907480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:33.907465 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-openshift-ca\") pod \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\" (UID: \"7a7ad224-962d-45ab-b0ca-96c5240a14bf\") " Oct 30 06:39:33.907806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:33.907781 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-openshift-ca" pod "7a7ad224-962d-45ab-b0ca-96c5240a14bf" (UID: "7a7ad224-962d-45ab-b0ca-96c5240a14bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:33.908188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:33.908163 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:33.909668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:33.909629 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-xks-ca" pod "7a7ad224-962d-45ab-b0ca-96c5240a14bf" (UID: "7a7ad224-962d-45ab-b0ca-96c5240a14bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:39:33.909974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:33.909951 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7ad224-962d-45ab-b0ca-96c5240a14bf-kube-api-access-qgpxg" pod "7a7ad224-962d-45ab-b0ca-96c5240a14bf" (UID: "7a7ad224-962d-45ab-b0ca-96c5240a14bf"). InnerVolumeSpecName "kube-api-access-qgpxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:34.009858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:34.009812 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7ad224-962d-45ab-b0ca-96c5240a14bf-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:34.009858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:34.009847 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgpxg\" (UniqueName: \"kubernetes.io/projected/7a7ad224-962d-45ab-b0ca-96c5240a14bf-kube-api-access-qgpxg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:34.310710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:34.310625 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:39:34.518714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:34.518685 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" Oct 30 06:39:34.522121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:34.518811 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j" event={"ID":"7a7ad224-962d-45ab-b0ca-96c5240a14bf","Type":"ContainerDied","Data":"4c9600a021826053fbb1d3dccb1e6e8c0dd72d144c035fc08cce40262dcadc83"} Oct 30 06:39:34.522121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:34.521499 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c9600a021826053fbb1d3dccb1e6e8c0dd72d144c035fc08cce40262dcadc83" Oct 30 06:39:38.270959 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:39:38.270909 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:39:42.007489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:42.005838 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf"] Oct 30 06:39:47.006293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.006239 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" podUID="23629bab-569f-4784-a3f5-b74113564d2e" containerName="payload-processing" containerID="cri-o://dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3" gracePeriod=25 Oct 30 06:39:47.524347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.524296 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:39:47.549427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.549394 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-plugins-config-volume\") pod \"23629bab-569f-4784-a3f5-b74113564d2e\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " Oct 30 06:39:47.549651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.549614 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-routing-overlay-volume\") pod \"23629bab-569f-4784-a3f5-b74113564d2e\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " Oct 30 06:39:47.549738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.549691 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-provider-credentials\") pod \"23629bab-569f-4784-a3f5-b74113564d2e\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " Oct 30 06:39:47.549738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.549723 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-kube-api-access-kv7xf\" (UniqueName: \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-kube-api-access-kv7xf\") pod \"23629bab-569f-4784-a3f5-b74113564d2e\" (UID: \"23629bab-569f-4784-a3f5-b74113564d2e\") " Oct 30 06:39:47.550014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.549981 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-plugins-config-volume" pod "23629bab-569f-4784-a3f5-b74113564d2e" (UID: "23629bab-569f-4784-a3f5-b74113564d2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:47.550159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.550046 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-routing-overlay-volume" pod "23629bab-569f-4784-a3f5-b74113564d2e" (UID: "23629bab-569f-4784-a3f5-b74113564d2e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:47.550493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.550440 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:47.550493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.550463 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23629bab-569f-4784-a3f5-b74113564d2e-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:47.552963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.552884 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-provider-credentials" pod "23629bab-569f-4784-a3f5-b74113564d2e" (UID: "23629bab-569f-4784-a3f5-b74113564d2e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:47.553089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.552991 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-kube-api-access-kv7xf" pod "23629bab-569f-4784-a3f5-b74113564d2e" (UID: "23629bab-569f-4784-a3f5-b74113564d2e"). InnerVolumeSpecName "kube-api-access-kv7xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:47.652170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.652131 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:47.652170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.652172 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv7xf\" (UniqueName: \"kubernetes.io/projected/23629bab-569f-4784-a3f5-b74113564d2e-kube-api-access-kv7xf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:39:47.876406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.876360 5001 generic.go:356] "Generic (PLEG): container finished" podID="23629bab-569f-4784-a3f5-b74113564d2e" containerID="dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3" exitCode=0 Oct 30 06:39:47.876616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.876448 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" Oct 30 06:39:47.876616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.876479 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" event={"ID":"23629bab-569f-4784-a3f5-b74113564d2e","Type":"ContainerDied","Data":"dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3"} Oct 30 06:39:47.876616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.876546 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf" event={"ID":"23629bab-569f-4784-a3f5-b74113564d2e","Type":"ContainerDied","Data":"ce2ddb3879568bd03fb68a5fced923df9e5f239737cbf029113e532d0f226f22"} Oct 30 06:39:47.876616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.876575 5001 scope.go:122] "RemoveContainer" containerID="dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3" Oct 30 06:39:47.890842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.890817 5001 scope.go:122] "RemoveContainer" containerID="dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3" Oct 30 06:39:47.891224 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:39:47.891194 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3\": container with ID starting with dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3 not found: ID does not exist" containerID="dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3" Oct 30 06:39:47.891297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.891237 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3"} err="failed to get container status \"dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3\": rpc error: code = NotFound desc = could not find container \"dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3\": container with ID starting with dc2c26aafa5cace24a57ff3a9b0387b7ba414ef07af3b5fa50243911c6bf61b3 not found: ID does not exist" Oct 30 06:39:47.935095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.935062 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf"] Oct 30 06:39:47.948412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:47.948379 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-cd7c22ggf"] Oct 30 06:39:48.124350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:48.124299 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23629bab-569f-4784-a3f5-b74113564d2e" path="/var/lib/kubelet/pods/23629bab-569f-4784-a3f5-b74113564d2e/volumes" Oct 30 06:39:57.073544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:57.073487 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:39:57.074070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:39:57.073596 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:40:02.468765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:02.468724 5001 generic.go:356] "Generic (PLEG): container finished" podID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerID="cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672" exitCode=0 Oct 30 06:40:02.469255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:02.468793 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" event={"ID":"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb","Type":"ContainerDied","Data":"cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672"} Oct 30 06:40:02.469400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:02.469382 5001 scope.go:122] "RemoveContainer" containerID="cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672" Oct 30 06:40:02.471849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:02.471815 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:40:02.472021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:02.471905 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:40:02.472021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:02.471970 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:40:03.483965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:03.483927 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" event={"ID":"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb","Type":"ContainerStarted","Data":"99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc"} Oct 30 06:40:04.171804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:04.171764 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:40:04.172253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:04.172160 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 30 06:40:04.172455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:04.172432 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 30 06:40:05.172331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:05.170793 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 30 06:40:05.172331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:05.170869 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 30 06:40:06.178586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:06.178070 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 30 06:40:06.178586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:06.178160 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 30 06:40:07.172232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:07.172197 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 30 06:40:07.176739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:07.176703 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 30 06:40:08.187417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:08.184431 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 30 06:40:08.187417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:08.184495 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 30 06:40:08.667342 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:40:08.666836 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:40:09.169825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:09.169782 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 30 06:40:09.170161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:09.170123 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 30 06:40:10.266638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:10.263925 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:40:10.266638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:10.266297 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:40:10.269673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:10.269649 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:40:10.993197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:10.993157 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5"] Oct 30 06:40:10.993550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:10.993517 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" podUID="c80da823-bfbb-4b6e-9d73-b959cd28288f" containerName="istio-proxy" containerID="cri-o://eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2" gracePeriod=30 Oct 30 06:40:11.002170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:11.002123 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv"] Oct 30 06:40:11.002453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:11.002405 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" podUID="a2602925-a5f8-4e4c-b6b7-ea06c8068aff" containerName="istio-proxy" containerID="cri-o://a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9" gracePeriod=30 Oct 30 06:40:16.356090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.356059 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:40:16.398094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.398057 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:40:16.408806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.408773 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-data\") pod \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " Oct 30 06:40:16.408974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.408823 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-podinfo\") pod \"c80da823-bfbb-4b6e-9d73-b959cd28288f\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " Oct 30 06:40:16.408974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.408894 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-kube-api-access-nf8f5\" (UniqueName: \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-kube-api-access-nf8f5\") pod \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " Oct 30 06:40:16.408974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.408932 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-envoy\") pod \"c80da823-bfbb-4b6e-9d73-b959cd28288f\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " Oct 30 06:40:16.409241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409003 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-podinfo\") pod \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " Oct 30 06:40:16.409241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409039 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-socket\") pod \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " Oct 30 06:40:16.409241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409112 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-certs\") pod \"c80da823-bfbb-4b6e-9d73-b959cd28288f\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " Oct 30 06:40:16.409241 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409191 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-kube-api-access-kx2sj\" (UniqueName: \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-kube-api-access-kx2sj\") pod \"c80da823-bfbb-4b6e-9d73-b959cd28288f\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " Oct 30 06:40:16.409488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409251 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-envoy\") pod \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " Oct 30 06:40:16.409488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409107 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-data" pod "a2602925-a5f8-4e4c-b6b7-ea06c8068aff" (UID: "a2602925-a5f8-4e4c-b6b7-ea06c8068aff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:40:16.409488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409330 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-socket" pod "a2602925-a5f8-4e4c-b6b7-ea06c8068aff" (UID: "a2602925-a5f8-4e4c-b6b7-ea06c8068aff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:40:16.409488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409408 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-credential-socket\") pod \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " Oct 30 06:40:16.409488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409430 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-certs" pod "c80da823-bfbb-4b6e-9d73-b959cd28288f" (UID: "c80da823-bfbb-4b6e-9d73-b959cd28288f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:40:16.409488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409454 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-credential-socket\") pod \"c80da823-bfbb-4b6e-9d73-b959cd28288f\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " Oct 30 06:40:16.409838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409503 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-token\" (UniqueName: \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-token\") pod \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " Oct 30 06:40:16.409838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409560 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istiod-ca-cert\") pod \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " Oct 30 06:40:16.409838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409602 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-data\") pod \"c80da823-bfbb-4b6e-9d73-b959cd28288f\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " Oct 30 06:40:16.409838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409672 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-certs\") pod \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\" (UID: \"a2602925-a5f8-4e4c-b6b7-ea06c8068aff\") " Oct 30 06:40:16.409838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409717 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-token\" (UniqueName: \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-token\") pod \"c80da823-bfbb-4b6e-9d73-b959cd28288f\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " Oct 30 06:40:16.409838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409769 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80da823-bfbb-4b6e-9d73-b959cd28288f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80da823-bfbb-4b6e-9d73-b959cd28288f-istiod-ca-cert\") pod \"c80da823-bfbb-4b6e-9d73-b959cd28288f\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " Oct 30 06:40:16.409838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409819 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-socket\") pod \"c80da823-bfbb-4b6e-9d73-b959cd28288f\" (UID: \"c80da823-bfbb-4b6e-9d73-b959cd28288f\") " Oct 30 06:40:16.410522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.409714 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-credential-socket" pod "a2602925-a5f8-4e4c-b6b7-ea06c8068aff" (UID: "a2602925-a5f8-4e4c-b6b7-ea06c8068aff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:40:16.410650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.410631 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-certs" pod "a2602925-a5f8-4e4c-b6b7-ea06c8068aff" (UID: "a2602925-a5f8-4e4c-b6b7-ea06c8068aff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:40:16.411113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.410717 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-socket" pod "c80da823-bfbb-4b6e-9d73-b959cd28288f" (UID: "c80da823-bfbb-4b6e-9d73-b959cd28288f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:40:16.411113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.410483 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-credential-socket" pod "c80da823-bfbb-4b6e-9d73-b959cd28288f" (UID: "c80da823-bfbb-4b6e-9d73-b959cd28288f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:40:16.411113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.410911 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.411113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.410931 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.411113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.410946 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.411113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.410960 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.411113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.410972 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.411113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.410986 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.411113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.411020 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-data" pod "c80da823-bfbb-4b6e-9d73-b959cd28288f" (UID: "c80da823-bfbb-4b6e-9d73-b959cd28288f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:40:16.411756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.411337 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80da823-bfbb-4b6e-9d73-b959cd28288f-istiod-ca-cert" pod "c80da823-bfbb-4b6e-9d73-b959cd28288f" (UID: "c80da823-bfbb-4b6e-9d73-b959cd28288f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:40:16.411756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.411337 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istiod-ca-cert" pod "a2602925-a5f8-4e4c-b6b7-ea06c8068aff" (UID: "a2602925-a5f8-4e4c-b6b7-ea06c8068aff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:40:16.413466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.413437 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-kube-api-access-kx2sj" pod "c80da823-bfbb-4b6e-9d73-b959cd28288f" (UID: "c80da823-bfbb-4b6e-9d73-b959cd28288f"). InnerVolumeSpecName "kube-api-access-kx2sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:40:16.413586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.413472 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-envoy" pod "a2602925-a5f8-4e4c-b6b7-ea06c8068aff" (UID: "a2602925-a5f8-4e4c-b6b7-ea06c8068aff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:40:16.413586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.413528 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-podinfo" pod "c80da823-bfbb-4b6e-9d73-b959cd28288f" (UID: "c80da823-bfbb-4b6e-9d73-b959cd28288f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:40:16.413586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.413535 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-token" pod "a2602925-a5f8-4e4c-b6b7-ea06c8068aff" (UID: "a2602925-a5f8-4e4c-b6b7-ea06c8068aff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:40:16.413812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.413791 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-podinfo" pod "a2602925-a5f8-4e4c-b6b7-ea06c8068aff" (UID: "a2602925-a5f8-4e4c-b6b7-ea06c8068aff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:40:16.413889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.413867 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-envoy" pod "c80da823-bfbb-4b6e-9d73-b959cd28288f" (UID: "c80da823-bfbb-4b6e-9d73-b959cd28288f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:40:16.414214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.414189 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-token" pod "c80da823-bfbb-4b6e-9d73-b959cd28288f" (UID: "c80da823-bfbb-4b6e-9d73-b959cd28288f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:40:16.414358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.414335 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-kube-api-access-nf8f5" pod "a2602925-a5f8-4e4c-b6b7-ea06c8068aff" (UID: "a2602925-a5f8-4e4c-b6b7-ea06c8068aff"). InnerVolumeSpecName "kube-api-access-nf8f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:40:16.512299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512266 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx2sj\" (UniqueName: \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-kube-api-access-kx2sj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.512507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512299 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.512507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512356 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.512507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512364 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.512507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512372 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.512507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512380 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.512507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512387 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.512507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512395 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80da823-bfbb-4b6e-9d73-b959cd28288f-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.512507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512404 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.512507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512412 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf8f5\" (UniqueName: \"kubernetes.io/projected/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-kube-api-access-nf8f5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.512507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512420 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80da823-bfbb-4b6e-9d73-b959cd28288f-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.512507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.512428 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2602925-a5f8-4e4c-b6b7-ea06c8068aff-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:16.917549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.917451 5001 generic.go:356] "Generic (PLEG): container finished" podID="c80da823-bfbb-4b6e-9d73-b959cd28288f" containerID="eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2" exitCode=0 Oct 30 06:40:16.917770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.917618 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" event={"ID":"c80da823-bfbb-4b6e-9d73-b959cd28288f","Type":"ContainerDied","Data":"eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2"} Oct 30 06:40:16.917770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.917668 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" event={"ID":"c80da823-bfbb-4b6e-9d73-b959cd28288f","Type":"ContainerDied","Data":"653e9437431446032508adc92b9c41cff6cd706d1847e7655d31cc129c2f930b"} Oct 30 06:40:16.917770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.917694 5001 scope.go:122] "RemoveContainer" containerID="eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2" Oct 30 06:40:16.918439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.917849 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5" Oct 30 06:40:16.921074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.921046 5001 generic.go:356] "Generic (PLEG): container finished" podID="a2602925-a5f8-4e4c-b6b7-ea06c8068aff" containerID="a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9" exitCode=0 Oct 30 06:40:16.921344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.921280 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" event={"ID":"a2602925-a5f8-4e4c-b6b7-ea06c8068aff","Type":"ContainerDied","Data":"a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9"} Oct 30 06:40:16.921344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.921330 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" Oct 30 06:40:16.921548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.921373 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv" event={"ID":"a2602925-a5f8-4e4c-b6b7-ea06c8068aff","Type":"ContainerDied","Data":"f45440975334c3d7602a3bd4de37a69ae5be8d2eb5f219bfde42773357f09f83"} Oct 30 06:40:16.943803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.943768 5001 scope.go:122] "RemoveContainer" containerID="eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2" Oct 30 06:40:16.944154 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:40:16.944128 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2\": container with ID starting with eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2 not found: ID does not exist" containerID="eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2" Oct 30 06:40:16.944238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.944168 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2"} err="failed to get container status \"eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2\": rpc error: code = NotFound desc = could not find container \"eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2\": container with ID starting with eaa1c62fec0f92bdf055879d72495aa942930aa1a937c2817e0154f501615fc2 not found: ID does not exist" Oct 30 06:40:16.944238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.944190 5001 scope.go:122] "RemoveContainer" containerID="a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9" Oct 30 06:40:16.956575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.956551 5001 scope.go:122] "RemoveContainer" containerID="a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9" Oct 30 06:40:16.956872 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:40:16.956846 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9\": container with ID starting with a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9 not found: ID does not exist" containerID="a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9" Oct 30 06:40:16.956949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.956885 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9"} err="failed to get container status \"a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9\": rpc error: code = NotFound desc = could not find container \"a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9\": container with ID starting with a74ff67039a1673753d86833b0e88b3d20d74f7308516bea6ea39e90c516bae9 not found: ID does not exist" Oct 30 06:40:16.998645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:16.998594 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5"] Oct 30 06:40:17.003899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:17.003868 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6hzf5"] Oct 30 06:40:17.037918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:17.037883 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv"] Oct 30 06:40:17.043029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:17.043000 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-9dmkv"] Oct 30 06:40:18.121369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:18.121334 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2602925-a5f8-4e4c-b6b7-ea06c8068aff" path="/var/lib/kubelet/pods/a2602925-a5f8-4e4c-b6b7-ea06c8068aff/volumes" Oct 30 06:40:18.121946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:18.121931 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80da823-bfbb-4b6e-9d73-b959cd28288f" path="/var/lib/kubelet/pods/c80da823-bfbb-4b6e-9d73-b959cd28288f/volumes" Oct 30 06:40:32.416077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:32.416039 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:40:37.949857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.949819 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk"] Oct 30 06:40:37.950742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950724 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c80da823-bfbb-4b6e-9d73-b959cd28288f" containerName="istio-proxy" Oct 30 06:40:37.950742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950742 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80da823-bfbb-4b6e-9d73-b959cd28288f" containerName="istio-proxy" Oct 30 06:40:37.950905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950769 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a2602925-a5f8-4e4c-b6b7-ea06c8068aff" containerName="istio-proxy" Oct 30 06:40:37.950905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950775 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2602925-a5f8-4e4c-b6b7-ea06c8068aff" containerName="istio-proxy" Oct 30 06:40:37.950905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950786 5001 cpu_manager.go:414] "containerMap: removing container" podUID="23629bab-569f-4784-a3f5-b74113564d2e" containerName="payload-processing" Oct 30 06:40:37.950905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950791 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23629bab-569f-4784-a3f5-b74113564d2e" containerName="payload-processing" Oct 30 06:40:37.950905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950801 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7a7ad224-962d-45ab-b0ca-96c5240a14bf" containerName="revoke-keys" Oct 30 06:40:37.950905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950806 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7ad224-962d-45ab-b0ca-96c5240a14bf" containerName="revoke-keys" Oct 30 06:40:37.951238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950930 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80da823-bfbb-4b6e-9d73-b959cd28288f" containerName="istio-proxy" Oct 30 06:40:37.951238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950942 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2602925-a5f8-4e4c-b6b7-ea06c8068aff" containerName="istio-proxy" Oct 30 06:40:37.951238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950950 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="23629bab-569f-4784-a3f5-b74113564d2e" containerName="payload-processing" Oct 30 06:40:37.951238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.950957 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7ad224-962d-45ab-b0ca-96c5240a14bf" containerName="revoke-keys" Oct 30 06:40:37.953992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.953972 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:37.974901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.974866 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk"] Oct 30 06:40:37.980486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.980452 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:40:37.980677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.980571 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:40:37.980677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.980619 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:40:37.982011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:37.981988 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:40:38.116767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:38.116729 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce51c758-47pzk\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:38.117116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:38.117099 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmzb\" (UniqueName: \"kubernetes.io/projected/85d58392-5b35-45c2-abd3-e8838e291c9f-kube-api-access-jqmzb\") pod \"maas-api-revoke-sub-ce51c758-47pzk\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:38.117182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:38.117168 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce51c758-47pzk\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:38.218367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:38.218255 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmzb\" (UniqueName: \"kubernetes.io/projected/85d58392-5b35-45c2-abd3-e8838e291c9f-kube-api-access-jqmzb\") pod \"maas-api-revoke-sub-ce51c758-47pzk\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:38.218561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:38.218397 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce51c758-47pzk\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:38.218634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:38.218597 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce51c758-47pzk\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:38.219277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:38.219246 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce51c758-47pzk\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:38.222058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:38.222033 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce51c758-47pzk\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:38.243515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:38.243489 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmzb\" (UniqueName: \"kubernetes.io/projected/85d58392-5b35-45c2-abd3-e8838e291c9f-kube-api-access-jqmzb\") pod \"maas-api-revoke-sub-ce51c758-47pzk\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:38.269814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:38.269762 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:38.273503 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:40:38.273471 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:40:38.626420 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:40:38.622973 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d58392_5b35_45c2_abd3_e8838e291c9f.slice/crio-02cb0ca9730d746bf3977e97e45950a0d58a69a15d7add0bd85560ef1fb860ab WatchSource:0}: Error finding container 02cb0ca9730d746bf3977e97e45950a0d58a69a15d7add0bd85560ef1fb860ab: Status 404 returned error can't find the container with id 02cb0ca9730d746bf3977e97e45950a0d58a69a15d7add0bd85560ef1fb860ab Oct 30 06:40:38.626420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:38.626252 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk"] Oct 30 06:40:39.395939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:39.395889 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" event={"ID":"85d58392-5b35-45c2-abd3-e8838e291c9f","Type":"ContainerStarted","Data":"68f498f84dca1e8b9447186a42da5b85a9fad8e377e4e18fa53be8852e70238d"} Oct 30 06:40:39.396423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:39.395965 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" event={"ID":"85d58392-5b35-45c2-abd3-e8838e291c9f","Type":"ContainerStarted","Data":"02cb0ca9730d746bf3977e97e45950a0d58a69a15d7add0bd85560ef1fb860ab"} Oct 30 06:40:39.440184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:39.440132 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" podStartSLOduration=2.440116111 podStartE2EDuration="2.440116111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:40:39.437648335 +0000 UTC m=+2853.433605345" watchObservedRunningTime="2026-10-30 06:40:39.440116111 +0000 UTC m=+2853.436073119" Oct 30 06:40:40.409137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:40.409086 5001 generic.go:356] "Generic (PLEG): container finished" podID="85d58392-5b35-45c2-abd3-e8838e291c9f" containerID="68f498f84dca1e8b9447186a42da5b85a9fad8e377e4e18fa53be8852e70238d" exitCode=0 Oct 30 06:40:40.409765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:40.409267 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" event={"ID":"85d58392-5b35-45c2-abd3-e8838e291c9f","Type":"ContainerDied","Data":"68f498f84dca1e8b9447186a42da5b85a9fad8e377e4e18fa53be8852e70238d"} Oct 30 06:40:41.633162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:41.633135 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:41.706953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:41.706859 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-openshift-ca\") pod \"85d58392-5b35-45c2-abd3-e8838e291c9f\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " Oct 30 06:40:41.706953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:41.706947 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-xks-ca\") pod \"85d58392-5b35-45c2-abd3-e8838e291c9f\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " Oct 30 06:40:41.707218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:41.707018 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d58392-5b35-45c2-abd3-e8838e291c9f-kube-api-access-jqmzb\" (UniqueName: \"kubernetes.io/projected/85d58392-5b35-45c2-abd3-e8838e291c9f-kube-api-access-jqmzb\") pod \"85d58392-5b35-45c2-abd3-e8838e291c9f\" (UID: \"85d58392-5b35-45c2-abd3-e8838e291c9f\") " Oct 30 06:40:41.707343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:41.707296 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-openshift-ca" pod "85d58392-5b35-45c2-abd3-e8838e291c9f" (UID: "85d58392-5b35-45c2-abd3-e8838e291c9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:40:41.707991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:41.707967 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:41.709850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:41.709816 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-xks-ca" pod "85d58392-5b35-45c2-abd3-e8838e291c9f" (UID: "85d58392-5b35-45c2-abd3-e8838e291c9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:40:41.710107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:41.710085 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d58392-5b35-45c2-abd3-e8838e291c9f-kube-api-access-jqmzb" pod "85d58392-5b35-45c2-abd3-e8838e291c9f" (UID: "85d58392-5b35-45c2-abd3-e8838e291c9f"). InnerVolumeSpecName "kube-api-access-jqmzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:40:41.809760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:41.809722 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d58392-5b35-45c2-abd3-e8838e291c9f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:41.809760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:41.809752 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmzb\" (UniqueName: \"kubernetes.io/projected/85d58392-5b35-45c2-abd3-e8838e291c9f-kube-api-access-jqmzb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:42.429199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:42.429102 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" event={"ID":"85d58392-5b35-45c2-abd3-e8838e291c9f","Type":"ContainerDied","Data":"02cb0ca9730d746bf3977e97e45950a0d58a69a15d7add0bd85560ef1fb860ab"} Oct 30 06:40:42.429199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:42.429140 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk" Oct 30 06:40:42.429199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:42.429158 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02cb0ca9730d746bf3977e97e45950a0d58a69a15d7add0bd85560ef1fb860ab" Oct 30 06:40:48.680294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.680256 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc"] Oct 30 06:40:48.681395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.681372 5001 cpu_manager.go:414] "containerMap: removing container" podUID="85d58392-5b35-45c2-abd3-e8838e291c9f" containerName="revoke-keys" Oct 30 06:40:48.681395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.681394 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d58392-5b35-45c2-abd3-e8838e291c9f" containerName="revoke-keys" Oct 30 06:40:48.681609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.681595 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d58392-5b35-45c2-abd3-e8838e291c9f" containerName="revoke-keys" Oct 30 06:40:48.684791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.684769 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:48.705339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.705288 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc"] Oct 30 06:40:48.714378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.710868 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:40:48.714378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.710881 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:40:48.714378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.710908 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:40:48.714378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.710974 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:40:48.853955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.853913 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03687e57-8kfhc\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:48.854249 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.854225 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03687e57-8kfhc\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:48.854374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.854257 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48zz\" (UniqueName: \"kubernetes.io/projected/4c018948-98ed-4cbd-afd6-6f26e63908de-kube-api-access-g48zz\") pod \"maas-api-revoke-sub-03687e57-8kfhc\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:48.955954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.955850 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03687e57-8kfhc\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:48.955954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.955894 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48zz\" (UniqueName: \"kubernetes.io/projected/4c018948-98ed-4cbd-afd6-6f26e63908de-kube-api-access-g48zz\") pod \"maas-api-revoke-sub-03687e57-8kfhc\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:48.956223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.956031 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03687e57-8kfhc\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:48.956606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.956572 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03687e57-8kfhc\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:48.958850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.958820 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03687e57-8kfhc\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:48.980430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:48.980395 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48zz\" (UniqueName: \"kubernetes.io/projected/4c018948-98ed-4cbd-afd6-6f26e63908de-kube-api-access-g48zz\") pod \"maas-api-revoke-sub-03687e57-8kfhc\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:49.001102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:49.001061 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:49.325653 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:40:49.325620 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c018948_98ed_4cbd_afd6_6f26e63908de.slice/crio-999f0b2c7d1d5f7a285d50532633638bc16151c24090a78715e0101037acde67 WatchSource:0}: Error finding container 999f0b2c7d1d5f7a285d50532633638bc16151c24090a78715e0101037acde67: Status 404 returned error can't find the container with id 999f0b2c7d1d5f7a285d50532633638bc16151c24090a78715e0101037acde67 Oct 30 06:40:49.328744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:49.328720 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc"] Oct 30 06:40:49.544473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:49.544436 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" event={"ID":"4c018948-98ed-4cbd-afd6-6f26e63908de","Type":"ContainerStarted","Data":"999f0b2c7d1d5f7a285d50532633638bc16151c24090a78715e0101037acde67"} Oct 30 06:40:50.557735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:50.557684 5001 generic.go:356] "Generic (PLEG): container finished" podID="4c018948-98ed-4cbd-afd6-6f26e63908de" containerID="6d6421e8f481c65db6d18e6af145d5cfbf9dee6b700be6925f7f56ab438d4571" exitCode=0 Oct 30 06:40:50.558271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:50.557857 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" event={"ID":"4c018948-98ed-4cbd-afd6-6f26e63908de","Type":"ContainerDied","Data":"6d6421e8f481c65db6d18e6af145d5cfbf9dee6b700be6925f7f56ab438d4571"} Oct 30 06:40:51.882557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:51.882531 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:51.923981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:51.923925 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-xks-ca\") pod \"4c018948-98ed-4cbd-afd6-6f26e63908de\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " Oct 30 06:40:51.924296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:51.924207 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-openshift-ca\") pod \"4c018948-98ed-4cbd-afd6-6f26e63908de\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " Oct 30 06:40:51.924517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:51.924369 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c018948-98ed-4cbd-afd6-6f26e63908de-kube-api-access-g48zz\" (UniqueName: \"kubernetes.io/projected/4c018948-98ed-4cbd-afd6-6f26e63908de-kube-api-access-g48zz\") pod \"4c018948-98ed-4cbd-afd6-6f26e63908de\" (UID: \"4c018948-98ed-4cbd-afd6-6f26e63908de\") " Oct 30 06:40:51.925023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:51.924854 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-openshift-ca" pod "4c018948-98ed-4cbd-afd6-6f26e63908de" (UID: "4c018948-98ed-4cbd-afd6-6f26e63908de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:40:51.925216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:51.925194 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:51.927842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:51.927800 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c018948-98ed-4cbd-afd6-6f26e63908de-kube-api-access-g48zz" pod "4c018948-98ed-4cbd-afd6-6f26e63908de" (UID: "4c018948-98ed-4cbd-afd6-6f26e63908de"). InnerVolumeSpecName "kube-api-access-g48zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:40:51.928696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:51.928673 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-xks-ca" pod "4c018948-98ed-4cbd-afd6-6f26e63908de" (UID: "4c018948-98ed-4cbd-afd6-6f26e63908de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:40:52.027613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:52.027571 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c018948-98ed-4cbd-afd6-6f26e63908de-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:52.027613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:52.027616 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g48zz\" (UniqueName: \"kubernetes.io/projected/4c018948-98ed-4cbd-afd6-6f26e63908de-kube-api-access-g48zz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:40:52.579272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:52.579192 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" Oct 30 06:40:52.579272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:52.579204 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc" event={"ID":"4c018948-98ed-4cbd-afd6-6f26e63908de","Type":"ContainerDied","Data":"999f0b2c7d1d5f7a285d50532633638bc16151c24090a78715e0101037acde67"} Oct 30 06:40:52.579272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:40:52.579254 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="999f0b2c7d1d5f7a285d50532633638bc16151c24090a78715e0101037acde67" Oct 30 06:41:00.291024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.290982 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq"] Oct 30 06:41:00.291923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.291898 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4c018948-98ed-4cbd-afd6-6f26e63908de" containerName="revoke-keys" Oct 30 06:41:00.291923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.291923 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c018948-98ed-4cbd-afd6-6f26e63908de" containerName="revoke-keys" Oct 30 06:41:00.292059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.292046 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c018948-98ed-4cbd-afd6-6f26e63908de" containerName="revoke-keys" Oct 30 06:41:00.294981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.294964 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.315158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.315132 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq"] Oct 30 06:41:00.322185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.322155 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"agc-cred-4b4eb8b1d03a\"" Oct 30 06:41:00.322483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.322170 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b8ca49\"" Oct 30 06:41:00.322630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.322171 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"payload-processing-external-model-e2e--7e273d00-dockercfg-t26z8\"" Oct 30 06:41:00.322630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.322235 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"routing-overlay\"" Oct 30 06:41:00.322630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.322290 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"openshift-service-ca.crt\"" Oct 30 06:41:00.324856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.322352 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"kube-root-ca.crt\"" Oct 30 06:41:00.382247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.382213 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.382455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.382273 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2tsh\" (UniqueName: \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-kube-api-access-n2tsh\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.382557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.382534 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.382646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.382635 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.484437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.484379 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.484634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.484462 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.484634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.484507 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2tsh\" (UniqueName: \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-kube-api-access-n2tsh\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.484715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.484680 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.485274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.485242 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.485414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.485353 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.487469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.487442 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.509626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.509591 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2tsh\" (UniqueName: \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-kube-api-access-n2tsh\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.615849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.610992 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:00.945898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:00.945870 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq"] Oct 30 06:41:00.947130 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:41:00.946823 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59f9e95_6955_478d_90b1_b575231907fb.slice/crio-29fa67593b0abd925882090d93e2362525586725e8a230d0e84a11ec44ac905f WatchSource:0}: Error finding container 29fa67593b0abd925882090d93e2362525586725e8a230d0e84a11ec44ac905f: Status 404 returned error can't find the container with id 29fa67593b0abd925882090d93e2362525586725e8a230d0e84a11ec44ac905f Oct 30 06:41:01.005699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:01.005463 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq"] Oct 30 06:41:01.701257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:01.701219 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" event={"ID":"b59f9e95-6955-478d-90b1-b575231907fb","Type":"ContainerStarted","Data":"2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe"} Oct 30 06:41:01.701767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:01.701283 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:01.701767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:01.701319 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" event={"ID":"b59f9e95-6955-478d-90b1-b575231907fb","Type":"ContainerStarted","Data":"29fa67593b0abd925882090d93e2362525586725e8a230d0e84a11ec44ac905f"} Oct 30 06:41:01.744462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:01.744406 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" podStartSLOduration=1.744391501 podStartE2EDuration="1.744391501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:41:01.743139164 +0000 UTC m=+2875.739096174" watchObservedRunningTime="2026-10-30 06:41:01.744391501 +0000 UTC m=+2875.740348510" Oct 30 06:41:05.513580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:05.513396 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95"] Oct 30 06:41:05.514240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:05.513651 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" containerID="cri-o://341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6" gracePeriod=30 Oct 30 06:41:06.701480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:06.701428 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" podUID="b59f9e95-6955-478d-90b1-b575231907fb" containerName="payload-processing" containerID="cri-o://2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe" gracePeriod=25 Oct 30 06:41:07.020508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.020479 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:07.134874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.134841 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-kube-api-access-n2tsh\" (UniqueName: \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-kube-api-access-n2tsh\") pod \"b59f9e95-6955-478d-90b1-b575231907fb\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " Oct 30 06:41:07.135090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.135072 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-provider-credentials\") pod \"b59f9e95-6955-478d-90b1-b575231907fb\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " Oct 30 06:41:07.135185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.135170 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-routing-overlay-volume\") pod \"b59f9e95-6955-478d-90b1-b575231907fb\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " Oct 30 06:41:07.135254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.135214 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-plugins-config-volume\") pod \"b59f9e95-6955-478d-90b1-b575231907fb\" (UID: \"b59f9e95-6955-478d-90b1-b575231907fb\") " Oct 30 06:41:07.135606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.135585 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-routing-overlay-volume" pod "b59f9e95-6955-478d-90b1-b575231907fb" (UID: "b59f9e95-6955-478d-90b1-b575231907fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:07.136137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.136114 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-plugins-config-volume" pod "b59f9e95-6955-478d-90b1-b575231907fb" (UID: "b59f9e95-6955-478d-90b1-b575231907fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:07.136541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.136520 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:07.137939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.137913 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-kube-api-access-n2tsh" pod "b59f9e95-6955-478d-90b1-b575231907fb" (UID: "b59f9e95-6955-478d-90b1-b575231907fb"). InnerVolumeSpecName "kube-api-access-n2tsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:07.138122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.138095 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-provider-credentials" pod "b59f9e95-6955-478d-90b1-b575231907fb" (UID: "b59f9e95-6955-478d-90b1-b575231907fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:07.238225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.238139 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:07.238225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.238171 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b59f9e95-6955-478d-90b1-b575231907fb-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:07.238225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.238181 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2tsh\" (UniqueName: \"kubernetes.io/projected/b59f9e95-6955-478d-90b1-b575231907fb-kube-api-access-n2tsh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:07.792382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.792348 5001 generic.go:356] "Generic (PLEG): container finished" podID="b59f9e95-6955-478d-90b1-b575231907fb" containerID="2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe" exitCode=0 Oct 30 06:41:07.792860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.792428 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" Oct 30 06:41:07.792860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.792439 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" event={"ID":"b59f9e95-6955-478d-90b1-b575231907fb","Type":"ContainerDied","Data":"2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe"} Oct 30 06:41:07.792860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.792527 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq" event={"ID":"b59f9e95-6955-478d-90b1-b575231907fb","Type":"ContainerDied","Data":"29fa67593b0abd925882090d93e2362525586725e8a230d0e84a11ec44ac905f"} Oct 30 06:41:07.792860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.792555 5001 scope.go:122] "RemoveContainer" containerID="2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe" Oct 30 06:41:07.807008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.806989 5001 scope.go:122] "RemoveContainer" containerID="2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe" Oct 30 06:41:07.807341 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:07.807316 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe\": container with ID starting with 2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe not found: ID does not exist" containerID="2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe" Oct 30 06:41:07.807421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.807354 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe"} err="failed to get container status \"2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe\": rpc error: code = NotFound desc = could not find container \"2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe\": container with ID starting with 2b6a35b1f4e4b97a07cff294c5d71a870ad7e495c8ffd64fde4b592a72e195fe not found: ID does not exist" Oct 30 06:41:07.850544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.850511 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq"] Oct 30 06:41:07.863508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:07.863470 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-7c47tdtcq"] Oct 30 06:41:08.167790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:08.167756 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b59f9e95-6955-478d-90b1-b575231907fb" path="/var/lib/kubelet/pods/b59f9e95-6955-478d-90b1-b575231907fb/volumes" Oct 30 06:41:08.268200 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:08.268152 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:41:10.822529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.822503 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:41:10.825132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.825106 5001 generic.go:356] "Generic (PLEG): container finished" podID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerID="341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6" exitCode=0 Oct 30 06:41:10.825279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.825140 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" event={"ID":"2fc22a5c-aff0-46dc-be59-b438bffb07a9","Type":"ContainerDied","Data":"341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6"} Oct 30 06:41:10.825279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.825162 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" Oct 30 06:41:10.825279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.825212 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95" event={"ID":"2fc22a5c-aff0-46dc-be59-b438bffb07a9","Type":"ContainerDied","Data":"0b4e0eb7a9642f0ffe1560b9fd60b5b8f8c78e1f312248318afd29f8a4be3a01"} Oct 30 06:41:10.825279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.825238 5001 scope.go:122] "RemoveContainer" containerID="341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6" Oct 30 06:41:10.841965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.841945 5001 scope.go:122] "RemoveContainer" containerID="7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817" Oct 30 06:41:10.853702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.853672 5001 scope.go:122] "RemoveContainer" containerID="341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6" Oct 30 06:41:10.854038 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:10.854008 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6\": container with ID starting with 341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6 not found: ID does not exist" containerID="341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6" Oct 30 06:41:10.854111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.854039 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6"} err="failed to get container status \"341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6\": rpc error: code = NotFound desc = could not find container \"341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6\": container with ID starting with 341603c22e048802a0f1b7142d19f9ccd47ff63983c832ef758c0372ac726bc6 not found: ID does not exist" Oct 30 06:41:10.854111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.854057 5001 scope.go:122] "RemoveContainer" containerID="7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817" Oct 30 06:41:10.854286 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:10.854271 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817\": container with ID starting with 7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817 not found: ID does not exist" containerID="7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817" Oct 30 06:41:10.854352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.854291 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817"} err="failed to get container status \"7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817\": rpc error: code = NotFound desc = could not find container \"7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817\": container with ID starting with 7a2405c6cbf605554c5eb62de8ef892452c7b4ad835cf2214047041be41e2817 not found: ID does not exist" Oct 30 06:41:10.903641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.903615 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-certs\") pod \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " Oct 30 06:41:10.903716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.903675 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-envoy\") pod \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " Oct 30 06:41:10.903780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.903747 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-token\" (UniqueName: \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-token\") pod \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " Oct 30 06:41:10.903916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.903805 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-credential-socket\") pod \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " Oct 30 06:41:10.903916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.903832 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-socket\") pod \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " Oct 30 06:41:10.904042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.903932 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-certs" pod "2fc22a5c-aff0-46dc-be59-b438bffb07a9" (UID: "2fc22a5c-aff0-46dc-be59-b438bffb07a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:41:10.904042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.903963 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-kube-api-access-4qdmm\" (UniqueName: \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-kube-api-access-4qdmm\") pod \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " Oct 30 06:41:10.904042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904008 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-data\") pod \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " Oct 30 06:41:10.904197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904057 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-podinfo\") pod \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " Oct 30 06:41:10.904197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904128 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-credential-socket" pod "2fc22a5c-aff0-46dc-be59-b438bffb07a9" (UID: "2fc22a5c-aff0-46dc-be59-b438bffb07a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:41:10.904197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904164 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istiod-ca-cert\") pod \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\" (UID: \"2fc22a5c-aff0-46dc-be59-b438bffb07a9\") " Oct 30 06:41:10.904197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904185 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-socket" pod "2fc22a5c-aff0-46dc-be59-b438bffb07a9" (UID: "2fc22a5c-aff0-46dc-be59-b438bffb07a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:41:10.905009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904605 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-data" pod "2fc22a5c-aff0-46dc-be59-b438bffb07a9" (UID: "2fc22a5c-aff0-46dc-be59-b438bffb07a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:41:10.905009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904773 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istiod-ca-cert" pod "2fc22a5c-aff0-46dc-be59-b438bffb07a9" (UID: "2fc22a5c-aff0-46dc-be59-b438bffb07a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:10.905009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904939 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:10.905009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904952 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:10.905009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904963 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:10.905009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904972 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:10.905009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.904984 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:10.906989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.906963 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-kube-api-access-4qdmm" pod "2fc22a5c-aff0-46dc-be59-b438bffb07a9" (UID: "2fc22a5c-aff0-46dc-be59-b438bffb07a9"). InnerVolumeSpecName "kube-api-access-4qdmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:10.907259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.907234 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-envoy" pod "2fc22a5c-aff0-46dc-be59-b438bffb07a9" (UID: "2fc22a5c-aff0-46dc-be59-b438bffb07a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:41:10.907456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.907293 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-podinfo" pod "2fc22a5c-aff0-46dc-be59-b438bffb07a9" (UID: "2fc22a5c-aff0-46dc-be59-b438bffb07a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:41:10.907940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:10.907903 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-token" pod "2fc22a5c-aff0-46dc-be59-b438bffb07a9" (UID: "2fc22a5c-aff0-46dc-be59-b438bffb07a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:11.011266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:11.011152 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:11.011266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:11.011191 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:11.011266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:11.011206 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qdmm\" (UniqueName: \"kubernetes.io/projected/2fc22a5c-aff0-46dc-be59-b438bffb07a9-kube-api-access-4qdmm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:11.011266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:11.011223 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc22a5c-aff0-46dc-be59-b438bffb07a9-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:11.178976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:11.178932 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95"] Oct 30 06:41:11.194866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:11.194828 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-z4n95"] Oct 30 06:41:12.001417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:12.001376 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:41:12.125608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:12.125567 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" path="/var/lib/kubelet/pods/2fc22a5c-aff0-46dc-be59-b438bffb07a9/volumes" Oct 30 06:41:15.236480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.236442 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm"] Oct 30 06:41:15.237292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.237271 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" Oct 30 06:41:15.237370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.237294 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" Oct 30 06:41:15.237370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.237327 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b59f9e95-6955-478d-90b1-b575231907fb" containerName="payload-processing" Oct 30 06:41:15.237370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.237333 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b59f9e95-6955-478d-90b1-b575231907fb" containerName="payload-processing" Oct 30 06:41:15.237496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.237485 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" Oct 30 06:41:15.237533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.237501 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b59f9e95-6955-478d-90b1-b575231907fb" containerName="payload-processing" Oct 30 06:41:15.237533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.237510 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" Oct 30 06:41:15.245806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.245770 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:15.264109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.264073 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm"] Oct 30 06:41:15.278718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.268169 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:41:15.278718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.268230 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:41:15.278718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.268252 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:41:15.278718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.268269 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:41:15.345039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.344983 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcec73f4-j2fmm\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:15.345378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.345343 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pmn\" (UniqueName: \"kubernetes.io/projected/d6675be1-bb85-40d1-85ed-0ec1c78825c0-kube-api-access-x4pmn\") pod \"maas-api-revoke-sub-fcec73f4-j2fmm\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:15.345532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.345440 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcec73f4-j2fmm\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:15.460332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.449718 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pmn\" (UniqueName: \"kubernetes.io/projected/d6675be1-bb85-40d1-85ed-0ec1c78825c0-kube-api-access-x4pmn\") pod \"maas-api-revoke-sub-fcec73f4-j2fmm\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:15.460332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.449817 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcec73f4-j2fmm\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:15.460332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.449991 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcec73f4-j2fmm\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:15.460332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.458186 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcec73f4-j2fmm\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:15.507803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.487247 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcec73f4-j2fmm\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:15.508081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.507172 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pmn\" (UniqueName: \"kubernetes.io/projected/d6675be1-bb85-40d1-85ed-0ec1c78825c0-kube-api-access-x4pmn\") pod \"maas-api-revoke-sub-fcec73f4-j2fmm\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:15.566636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.566604 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:15.891747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.891714 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm"] Oct 30 06:41:15.892636 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:41:15.892607 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6675be1_bb85_40d1_85ed_0ec1c78825c0.slice/crio-bc4e508b48d9547d513b425e6cf08ac7285601d747965ffc5ca5e0606ea9a7e7 WatchSource:0}: Error finding container bc4e508b48d9547d513b425e6cf08ac7285601d747965ffc5ca5e0606ea9a7e7: Status 404 returned error can't find the container with id bc4e508b48d9547d513b425e6cf08ac7285601d747965ffc5ca5e0606ea9a7e7 Oct 30 06:41:15.957386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:15.957349 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" event={"ID":"d6675be1-bb85-40d1-85ed-0ec1c78825c0","Type":"ContainerStarted","Data":"bc4e508b48d9547d513b425e6cf08ac7285601d747965ffc5ca5e0606ea9a7e7"} Oct 30 06:41:16.128029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.127998 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt"] Oct 30 06:41:16.129132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.129103 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" Oct 30 06:41:16.129132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.129133 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc22a5c-aff0-46dc-be59-b438bffb07a9" containerName="istio-proxy" Oct 30 06:41:16.137569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.137546 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:16.153264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.153188 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt"] Oct 30 06:41:16.267743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.267717 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c15d567-bpjnt\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:16.268132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.267891 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c15d567-bpjnt\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:16.268132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.267979 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89chq\" (UniqueName: \"kubernetes.io/projected/663956cb-4785-4eed-b392-9764d9f52c20-kube-api-access-89chq\") pod \"maas-api-revoke-sub-1c15d567-bpjnt\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:16.370097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.370062 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c15d567-bpjnt\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:16.370334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.370149 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c15d567-bpjnt\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:16.370334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.370178 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89chq\" (UniqueName: \"kubernetes.io/projected/663956cb-4785-4eed-b392-9764d9f52c20-kube-api-access-89chq\") pod \"maas-api-revoke-sub-1c15d567-bpjnt\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:16.370846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.370821 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c15d567-bpjnt\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:16.372977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.372952 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c15d567-bpjnt\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:16.394651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.394628 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89chq\" (UniqueName: \"kubernetes.io/projected/663956cb-4785-4eed-b392-9764d9f52c20-kube-api-access-89chq\") pod \"maas-api-revoke-sub-1c15d567-bpjnt\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:16.467946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.467855 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:16.807883 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:41:16.807846 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663956cb_4785_4eed_b392_9764d9f52c20.slice/crio-3725eb225cce201069f248b287d659d1e228d2b3eb3a17fa39fa31c3f18bbbad WatchSource:0}: Error finding container 3725eb225cce201069f248b287d659d1e228d2b3eb3a17fa39fa31c3f18bbbad: Status 404 returned error can't find the container with id 3725eb225cce201069f248b287d659d1e228d2b3eb3a17fa39fa31c3f18bbbad Oct 30 06:41:16.808204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.808179 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt"] Oct 30 06:41:16.866269 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.866231 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl"] Oct 30 06:41:16.872762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.872739 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:16.889421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.889394 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4edf35b-fq6xl\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:16.889590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.889536 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcp82\" (UniqueName: \"kubernetes.io/projected/301b496f-7f03-4b61-be50-9648294b3b3c-kube-api-access-pcp82\") pod \"maas-api-revoke-sub-d4edf35b-fq6xl\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:16.889741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.889727 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4edf35b-fq6xl\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:16.892191 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.892162 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl"] Oct 30 06:41:16.971203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.971166 5001 generic.go:356] "Generic (PLEG): container finished" podID="d6675be1-bb85-40d1-85ed-0ec1c78825c0" containerID="cf1e468a0806872eb07ddd7bb49cb2f3bf3fa2df5281660072cf387a0556b076" exitCode=0 Oct 30 06:41:16.971593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.971289 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" event={"ID":"d6675be1-bb85-40d1-85ed-0ec1c78825c0","Type":"ContainerDied","Data":"cf1e468a0806872eb07ddd7bb49cb2f3bf3fa2df5281660072cf387a0556b076"} Oct 30 06:41:16.973294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.973266 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" event={"ID":"663956cb-4785-4eed-b392-9764d9f52c20","Type":"ContainerStarted","Data":"3725eb225cce201069f248b287d659d1e228d2b3eb3a17fa39fa31c3f18bbbad"} Oct 30 06:41:16.991517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.991483 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcp82\" (UniqueName: \"kubernetes.io/projected/301b496f-7f03-4b61-be50-9648294b3b3c-kube-api-access-pcp82\") pod \"maas-api-revoke-sub-d4edf35b-fq6xl\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:16.991751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.991734 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4edf35b-fq6xl\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:16.991829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.991782 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4edf35b-fq6xl\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:16.992766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:16.992738 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4edf35b-fq6xl\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:17.052768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:17.052722 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4edf35b-fq6xl\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:17.053095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:17.053068 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcp82\" (UniqueName: \"kubernetes.io/projected/301b496f-7f03-4b61-be50-9648294b3b3c-kube-api-access-pcp82\") pod \"maas-api-revoke-sub-d4edf35b-fq6xl\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:17.233380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:17.233346 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:17.565206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:17.564748 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl"] Oct 30 06:41:17.566009 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:41:17.565959 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301b496f_7f03_4b61_be50_9648294b3b3c.slice/crio-415c5bea6276c374488078be3bc568fe89da763b727698b8d571ffd432b96423 WatchSource:0}: Error finding container 415c5bea6276c374488078be3bc568fe89da763b727698b8d571ffd432b96423: Status 404 returned error can't find the container with id 415c5bea6276c374488078be3bc568fe89da763b727698b8d571ffd432b96423 Oct 30 06:41:17.986988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:17.986952 5001 generic.go:356] "Generic (PLEG): container finished" podID="663956cb-4785-4eed-b392-9764d9f52c20" containerID="254162814ccc56ae7fef298568305be7c98276912e2082f1f793133126f88b85" exitCode=0 Oct 30 06:41:17.987168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:17.987043 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" event={"ID":"663956cb-4785-4eed-b392-9764d9f52c20","Type":"ContainerDied","Data":"254162814ccc56ae7fef298568305be7c98276912e2082f1f793133126f88b85"} Oct 30 06:41:17.988686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:17.988657 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" event={"ID":"301b496f-7f03-4b61-be50-9648294b3b3c","Type":"ContainerStarted","Data":"415c5bea6276c374488078be3bc568fe89da763b727698b8d571ffd432b96423"} Oct 30 06:41:18.205526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:18.205499 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:18.215935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:18.215902 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6675be1-bb85-40d1-85ed-0ec1c78825c0-kube-api-access-x4pmn\" (UniqueName: \"kubernetes.io/projected/d6675be1-bb85-40d1-85ed-0ec1c78825c0-kube-api-access-x4pmn\") pod \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " Oct 30 06:41:18.216083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:18.215974 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-openshift-ca\") pod \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " Oct 30 06:41:18.216129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:18.216119 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-xks-ca\") pod \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\" (UID: \"d6675be1-bb85-40d1-85ed-0ec1c78825c0\") " Oct 30 06:41:18.216382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:18.216357 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-openshift-ca" pod "d6675be1-bb85-40d1-85ed-0ec1c78825c0" (UID: "d6675be1-bb85-40d1-85ed-0ec1c78825c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:18.216702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:18.216684 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:18.218286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:18.218265 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6675be1-bb85-40d1-85ed-0ec1c78825c0-kube-api-access-x4pmn" pod "d6675be1-bb85-40d1-85ed-0ec1c78825c0" (UID: "d6675be1-bb85-40d1-85ed-0ec1c78825c0"). InnerVolumeSpecName "kube-api-access-x4pmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:18.218401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:18.218293 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-xks-ca" pod "d6675be1-bb85-40d1-85ed-0ec1c78825c0" (UID: "d6675be1-bb85-40d1-85ed-0ec1c78825c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:41:18.318134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:18.318052 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6675be1-bb85-40d1-85ed-0ec1c78825c0-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:18.318134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:18.318081 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4pmn\" (UniqueName: \"kubernetes.io/projected/d6675be1-bb85-40d1-85ed-0ec1c78825c0-kube-api-access-x4pmn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:19.000369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.000295 5001 generic.go:356] "Generic (PLEG): container finished" podID="301b496f-7f03-4b61-be50-9648294b3b3c" containerID="66228458c3974ee2b7b7150e7b297c0a9bc42d998c824842d28c638fd0bd04a9" exitCode=0 Oct 30 06:41:19.000369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.000359 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" event={"ID":"301b496f-7f03-4b61-be50-9648294b3b3c","Type":"ContainerDied","Data":"66228458c3974ee2b7b7150e7b297c0a9bc42d998c824842d28c638fd0bd04a9"} Oct 30 06:41:19.002094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.002023 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" Oct 30 06:41:19.002094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.002063 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm" event={"ID":"d6675be1-bb85-40d1-85ed-0ec1c78825c0","Type":"ContainerDied","Data":"bc4e508b48d9547d513b425e6cf08ac7285601d747965ffc5ca5e0606ea9a7e7"} Oct 30 06:41:19.002285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.002100 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc4e508b48d9547d513b425e6cf08ac7285601d747965ffc5ca5e0606ea9a7e7" Oct 30 06:41:19.276380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.269498 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:19.355449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.355409 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-xks-ca\") pod \"663956cb-4785-4eed-b392-9764d9f52c20\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " Oct 30 06:41:19.355662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.355487 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663956cb-4785-4eed-b392-9764d9f52c20-kube-api-access-89chq\" (UniqueName: \"kubernetes.io/projected/663956cb-4785-4eed-b392-9764d9f52c20-kube-api-access-89chq\") pod \"663956cb-4785-4eed-b392-9764d9f52c20\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " Oct 30 06:41:19.355732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.355684 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-openshift-ca\") pod \"663956cb-4785-4eed-b392-9764d9f52c20\" (UID: \"663956cb-4785-4eed-b392-9764d9f52c20\") " Oct 30 06:41:19.356359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.356293 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-openshift-ca" pod "663956cb-4785-4eed-b392-9764d9f52c20" (UID: "663956cb-4785-4eed-b392-9764d9f52c20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:19.356530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.356508 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:19.358581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.358551 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-xks-ca" pod "663956cb-4785-4eed-b392-9764d9f52c20" (UID: "663956cb-4785-4eed-b392-9764d9f52c20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:41:19.358865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.358841 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663956cb-4785-4eed-b392-9764d9f52c20-kube-api-access-89chq" pod "663956cb-4785-4eed-b392-9764d9f52c20" (UID: "663956cb-4785-4eed-b392-9764d9f52c20"). InnerVolumeSpecName "kube-api-access-89chq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:19.458037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.457995 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663956cb-4785-4eed-b392-9764d9f52c20-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:19.458037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:19.458023 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89chq\" (UniqueName: \"kubernetes.io/projected/663956cb-4785-4eed-b392-9764d9f52c20-kube-api-access-89chq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:20.012299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.012264 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" Oct 30 06:41:20.012299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.012291 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt" event={"ID":"663956cb-4785-4eed-b392-9764d9f52c20","Type":"ContainerDied","Data":"3725eb225cce201069f248b287d659d1e228d2b3eb3a17fa39fa31c3f18bbbad"} Oct 30 06:41:20.012913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.012359 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3725eb225cce201069f248b287d659d1e228d2b3eb3a17fa39fa31c3f18bbbad" Oct 30 06:41:20.197419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.197392 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:20.271735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.271644 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-openshift-ca\") pod \"301b496f-7f03-4b61-be50-9648294b3b3c\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " Oct 30 06:41:20.271932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.271763 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-xks-ca\") pod \"301b496f-7f03-4b61-be50-9648294b3b3c\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " Oct 30 06:41:20.271932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.271839 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/301b496f-7f03-4b61-be50-9648294b3b3c-kube-api-access-pcp82\" (UniqueName: \"kubernetes.io/projected/301b496f-7f03-4b61-be50-9648294b3b3c-kube-api-access-pcp82\") pod \"301b496f-7f03-4b61-be50-9648294b3b3c\" (UID: \"301b496f-7f03-4b61-be50-9648294b3b3c\") " Oct 30 06:41:20.272108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.272081 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-openshift-ca" pod "301b496f-7f03-4b61-be50-9648294b3b3c" (UID: "301b496f-7f03-4b61-be50-9648294b3b3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:20.273326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.273279 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:20.275471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.274906 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-xks-ca" pod "301b496f-7f03-4b61-be50-9648294b3b3c" (UID: "301b496f-7f03-4b61-be50-9648294b3b3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:41:20.275956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.275927 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301b496f-7f03-4b61-be50-9648294b3b3c-kube-api-access-pcp82" pod "301b496f-7f03-4b61-be50-9648294b3b3c" (UID: "301b496f-7f03-4b61-be50-9648294b3b3c"). InnerVolumeSpecName "kube-api-access-pcp82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:20.375785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.375737 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301b496f-7f03-4b61-be50-9648294b3b3c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:20.375785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:20.375779 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcp82\" (UniqueName: \"kubernetes.io/projected/301b496f-7f03-4b61-be50-9648294b3b3c-kube-api-access-pcp82\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:21.023936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:21.023896 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" Oct 30 06:41:21.024392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:21.023942 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl" event={"ID":"301b496f-7f03-4b61-be50-9648294b3b3c","Type":"ContainerDied","Data":"415c5bea6276c374488078be3bc568fe89da763b727698b8d571ffd432b96423"} Oct 30 06:41:21.024392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:21.023991 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="415c5bea6276c374488078be3bc568fe89da763b727698b8d571ffd432b96423" Oct 30 06:41:23.563507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.563468 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-54cb654cf-98j9x"] Oct 30 06:41:23.564354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.564334 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d6675be1-bb85-40d1-85ed-0ec1c78825c0" containerName="revoke-keys" Oct 30 06:41:23.564354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.564353 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6675be1-bb85-40d1-85ed-0ec1c78825c0" containerName="revoke-keys" Oct 30 06:41:23.564529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.564376 5001 cpu_manager.go:414] "containerMap: removing container" podUID="663956cb-4785-4eed-b392-9764d9f52c20" containerName="revoke-keys" Oct 30 06:41:23.564529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.564382 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663956cb-4785-4eed-b392-9764d9f52c20" containerName="revoke-keys" Oct 30 06:41:23.564529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.564406 5001 cpu_manager.go:414] "containerMap: removing container" podUID="301b496f-7f03-4b61-be50-9648294b3b3c" containerName="revoke-keys" Oct 30 06:41:23.564529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.564412 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="301b496f-7f03-4b61-be50-9648294b3b3c" containerName="revoke-keys" Oct 30 06:41:23.564759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.564556 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6675be1-bb85-40d1-85ed-0ec1c78825c0" containerName="revoke-keys" Oct 30 06:41:23.564759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.564568 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="663956cb-4785-4eed-b392-9764d9f52c20" containerName="revoke-keys" Oct 30 06:41:23.564759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.564577 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="301b496f-7f03-4b61-be50-9648294b3b3c" containerName="revoke-keys" Oct 30 06:41:23.567819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.567796 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.590525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.590492 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54cb654cf-98j9x"] Oct 30 06:41:23.596375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.595296 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a6e5b91b8e65\"" Oct 30 06:41:23.694720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.694675 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-plugins-config-volume\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.694914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.694797 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-provider-credentials\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.695020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.694999 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9jq\" (UniqueName: \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-kube-api-access-rs9jq\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.695072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.695059 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-routing-overlay-volume\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.796905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.796868 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9jq\" (UniqueName: \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-kube-api-access-rs9jq\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.797159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.796932 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-routing-overlay-volume\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.797159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.797084 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-plugins-config-volume\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.797159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.797123 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-provider-credentials\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.797753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.797732 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-routing-overlay-volume\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.797890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.797868 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-plugins-config-volume\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.800298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.800268 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-provider-credentials\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.821497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.821431 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9jq\" (UniqueName: \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-kube-api-access-rs9jq\") pod \"payload-processing-external-model-54cb654cf-98j9x\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:23.885122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:23.885080 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:24.206112 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:41:24.206077 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d01dd55_de02_4f0c_83d6_303271866adf.slice/crio-f8c4e5452557cc4190342bb1368db140b4e7d7537406149713ae4d325ee53bd0 WatchSource:0}: Error finding container f8c4e5452557cc4190342bb1368db140b4e7d7537406149713ae4d325ee53bd0: Status 404 returned error can't find the container with id f8c4e5452557cc4190342bb1368db140b4e7d7537406149713ae4d325ee53bd0 Oct 30 06:41:24.206240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:24.206203 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54cb654cf-98j9x"] Oct 30 06:41:25.134283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:25.134245 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" event={"ID":"4d01dd55-de02-4f0c-83d6-303271866adf","Type":"ContainerStarted","Data":"88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c"} Oct 30 06:41:25.134832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:25.134346 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" event={"ID":"4d01dd55-de02-4f0c-83d6-303271866adf","Type":"ContainerStarted","Data":"f8c4e5452557cc4190342bb1368db140b4e7d7537406149713ae4d325ee53bd0"} Oct 30 06:41:25.134832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:25.134375 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:25.178487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:25.178428 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" podStartSLOduration=2.178412991 podStartE2EDuration="2.178412991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:41:25.176897083 +0000 UTC m=+2899.172854099" watchObservedRunningTime="2026-10-30 06:41:25.178412991 +0000 UTC m=+2899.174370017" Oct 30 06:41:27.356188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.356144 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj"] Oct 30 06:41:27.360270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.360247 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:27.382572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.382543 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj"] Oct 30 06:41:27.387535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.387496 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:41:27.387699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.387544 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:41:27.389001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.388968 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:41:27.389203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.389007 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:41:27.473012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.472974 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhfkc\" (UniqueName: \"kubernetes.io/projected/188927f8-697e-4b7f-9c6f-bd14ea0041f7-kube-api-access-rhfkc\") pod \"maas-api-revoke-sub-eeabfa6f-npczj\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:27.473198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.473033 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeabfa6f-npczj\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:27.473198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.473190 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeabfa6f-npczj\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:27.574727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.574691 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeabfa6f-npczj\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:27.575741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.575715 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhfkc\" (UniqueName: \"kubernetes.io/projected/188927f8-697e-4b7f-9c6f-bd14ea0041f7-kube-api-access-rhfkc\") pod \"maas-api-revoke-sub-eeabfa6f-npczj\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:27.575940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.575925 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeabfa6f-npczj\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:27.577169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.577144 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeabfa6f-npczj\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:27.581873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.581851 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeabfa6f-npczj\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:27.618473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.618390 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhfkc\" (UniqueName: \"kubernetes.io/projected/188927f8-697e-4b7f-9c6f-bd14ea0041f7-kube-api-access-rhfkc\") pod \"maas-api-revoke-sub-eeabfa6f-npczj\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:27.675721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:27.675680 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:28.089388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:28.086573 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l"] Oct 30 06:41:28.219712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:28.219681 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb4fd4f-hzq2l"] Oct 30 06:41:28.219888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:28.219736 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj"] Oct 30 06:41:29.274332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:29.273337 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" event={"ID":"188927f8-697e-4b7f-9c6f-bd14ea0041f7","Type":"ContainerStarted","Data":"baa350eff7c141d661e6c4998685280505910a7c76b0792472b28a9dec179e71"} Oct 30 06:41:29.274332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:29.273508 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" event={"ID":"188927f8-697e-4b7f-9c6f-bd14ea0041f7","Type":"ContainerStarted","Data":"5e5f281c05dc0d0359f4af4ce28dd6b440796f58d9768645ec18140e7b935fe1"} Oct 30 06:41:29.368335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:29.367732 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" podStartSLOduration=2.367706586 podStartE2EDuration="2.367706586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:41:29.331844137 +0000 UTC m=+2903.327801147" watchObservedRunningTime="2026-10-30 06:41:29.367706586 +0000 UTC m=+2903.363663599" Oct 30 06:41:30.118332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:30.106385 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v"] Oct 30 06:41:30.187332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:30.186059 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd" path="/var/lib/kubelet/pods/16ff22bf-0ec1-40d9-9b33-bfb2aad4e4dd/volumes" Oct 30 06:41:30.230400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:30.230360 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8546bc59-7cw7v"] Oct 30 06:41:30.304711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:30.304675 5001 generic.go:356] "Generic (PLEG): container finished" podID="188927f8-697e-4b7f-9c6f-bd14ea0041f7" containerID="baa350eff7c141d661e6c4998685280505910a7c76b0792472b28a9dec179e71" exitCode=0 Oct 30 06:41:30.305377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:30.305346 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" event={"ID":"188927f8-697e-4b7f-9c6f-bd14ea0041f7","Type":"ContainerDied","Data":"baa350eff7c141d661e6c4998685280505910a7c76b0792472b28a9dec179e71"} Oct 30 06:41:31.521552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:31.521525 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:31.640216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:31.640183 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-openshift-ca\") pod \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " Oct 30 06:41:31.640429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:31.640410 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188927f8-697e-4b7f-9c6f-bd14ea0041f7-kube-api-access-rhfkc\" (UniqueName: \"kubernetes.io/projected/188927f8-697e-4b7f-9c6f-bd14ea0041f7-kube-api-access-rhfkc\") pod \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " Oct 30 06:41:31.640503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:31.640481 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-xks-ca\") pod \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\" (UID: \"188927f8-697e-4b7f-9c6f-bd14ea0041f7\") " Oct 30 06:41:31.640685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:31.640652 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-openshift-ca" pod "188927f8-697e-4b7f-9c6f-bd14ea0041f7" (UID: "188927f8-697e-4b7f-9c6f-bd14ea0041f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:31.641118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:31.641097 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:31.642737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:31.642714 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-xks-ca" pod "188927f8-697e-4b7f-9c6f-bd14ea0041f7" (UID: "188927f8-697e-4b7f-9c6f-bd14ea0041f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:41:31.642878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:31.642856 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188927f8-697e-4b7f-9c6f-bd14ea0041f7-kube-api-access-rhfkc" pod "188927f8-697e-4b7f-9c6f-bd14ea0041f7" (UID: "188927f8-697e-4b7f-9c6f-bd14ea0041f7"). InnerVolumeSpecName "kube-api-access-rhfkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:31.742271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:31.742232 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhfkc\" (UniqueName: \"kubernetes.io/projected/188927f8-697e-4b7f-9c6f-bd14ea0041f7-kube-api-access-rhfkc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:31.742271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:31.742264 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188927f8-697e-4b7f-9c6f-bd14ea0041f7-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:32.122694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:32.122653 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="531c7fc6-dd7a-429f-ae46-8b8054339b93" path="/var/lib/kubelet/pods/531c7fc6-dd7a-429f-ae46-8b8054339b93/volumes" Oct 30 06:41:32.326127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:32.326102 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" Oct 30 06:41:32.326334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:32.326185 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj" event={"ID":"188927f8-697e-4b7f-9c6f-bd14ea0041f7","Type":"ContainerDied","Data":"5e5f281c05dc0d0359f4af4ce28dd6b440796f58d9768645ec18140e7b935fe1"} Oct 30 06:41:32.326334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:32.326231 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e5f281c05dc0d0359f4af4ce28dd6b440796f58d9768645ec18140e7b935fe1" Oct 30 06:41:36.147323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:36.147270 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:41:36.281816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:36.281782 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h"] Oct 30 06:41:38.271024 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:38.270987 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:41:39.221527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.221480 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg"] Oct 30 06:41:39.222426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.222408 5001 cpu_manager.go:414] "containerMap: removing container" podUID="188927f8-697e-4b7f-9c6f-bd14ea0041f7" containerName="revoke-keys" Oct 30 06:41:39.222493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.222427 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188927f8-697e-4b7f-9c6f-bd14ea0041f7" containerName="revoke-keys" Oct 30 06:41:39.222625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.222615 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="188927f8-697e-4b7f-9c6f-bd14ea0041f7" containerName="revoke-keys" Oct 30 06:41:39.226462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.226441 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.232627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.232586 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf4rs\" (UniqueName: \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-kube-api-access-qf4rs\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.232804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.232642 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.232804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.232689 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.232927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.232882 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.250545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.250513 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg"] Oct 30 06:41:39.255277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.255249 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2d2923\"" Oct 30 06:41:39.255459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.255419 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"agc-cred-c32bb415fa0e\"" Oct 30 06:41:39.255605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.255581 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"routing-overlay\"" Oct 30 06:41:39.255838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.255660 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"openshift-service-ca.crt\"" Oct 30 06:41:39.255838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.255808 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"kube-root-ca.crt\"" Oct 30 06:41:39.260233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.260198 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"payload-processing-external-model-e2e--d83828a5-dockercfg-cqlg5\"" Oct 30 06:41:39.334560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.334522 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.335045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.334629 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf4rs\" (UniqueName: \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-kube-api-access-qf4rs\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.335045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.334653 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.335045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.334686 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.335417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.335295 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.335417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.335294 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.337516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.337493 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.360687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.360651 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf4rs\" (UniqueName: \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-kube-api-access-qf4rs\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.543353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.543262 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:39.873365 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:41:39.873327 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1665a272_058b_4a3b_8058_9c92c47476e1.slice/crio-f93ddb2a112a843ff6a789721af1adeab160f78ff46dae0f2c9c04fd8fe0ab9a WatchSource:0}: Error finding container f93ddb2a112a843ff6a789721af1adeab160f78ff46dae0f2c9c04fd8fe0ab9a: Status 404 returned error can't find the container with id f93ddb2a112a843ff6a789721af1adeab160f78ff46dae0f2c9c04fd8fe0ab9a Oct 30 06:41:39.875603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:39.875567 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg"] Oct 30 06:41:40.447504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:40.447465 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" event={"ID":"1665a272-058b-4a3b-8058-9c92c47476e1","Type":"ContainerStarted","Data":"1aecdd0de1dae5cbbea1837f3df9b7c269c9c425d6a4a7eab1af1f6fafa2d296"} Oct 30 06:41:40.448026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:40.447528 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:40.448026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:40.447546 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" event={"ID":"1665a272-058b-4a3b-8058-9c92c47476e1","Type":"ContainerStarted","Data":"f93ddb2a112a843ff6a789721af1adeab160f78ff46dae0f2c9c04fd8fe0ab9a"} Oct 30 06:41:40.489948 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:40.489910 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d2923/agc-cred-c32bb415fa0e: secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:40.489948 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:40.489940 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg: secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:40.490226 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:40.490059 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials podName:1665a272-058b-4a3b-8058-9c92c47476e1 nodeName:}" failed. No retries permitted until 2026-10-30 06:41:40.989994741 +0000 UTC m=+2914.985951748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1") : secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:40.494783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:40.494728 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" podStartSLOduration=1.494709828 podStartE2EDuration="1.494709828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:41:40.489146413 +0000 UTC m=+2914.485103424" watchObservedRunningTime="2026-10-30 06:41:40.494709828 +0000 UTC m=+2914.490666842" Oct 30 06:41:40.510584 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:40.510549 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg"] Oct 30 06:41:41.022331 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:41.002403 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d2923/agc-cred-c32bb415fa0e: secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:41.022623 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:41.022597 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg: secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:41.022844 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:41.022831 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials podName:1665a272-058b-4a3b-8058-9c92c47476e1 nodeName:}" failed. No retries permitted until 2026-10-30 06:41:42.02280502 +0000 UTC m=+2916.018762025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1") : secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:41.283086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.282990 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" podUID="9aaf92ff-2a28-4d87-97e9-06e555adb83d" containerName="payload-processing" containerID="cri-o://d07b68051f807b268e9c61ef795c45d59cff3f5be51d2583aa46a7c00a9cf09b" gracePeriod=25 Oct 30 06:41:41.461981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.461947 5001 generic.go:356] "Generic (PLEG): container finished" podID="9aaf92ff-2a28-4d87-97e9-06e555adb83d" containerID="d07b68051f807b268e9c61ef795c45d59cff3f5be51d2583aa46a7c00a9cf09b" exitCode=0 Oct 30 06:41:41.462742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.462719 5001 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" secret="" err="secret \"payload-processing-external-model-e2e--d83828a5-dockercfg-cqlg5\" not found" Oct 30 06:41:41.463432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.463402 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" event={"ID":"9aaf92ff-2a28-4d87-97e9-06e555adb83d","Type":"ContainerDied","Data":"d07b68051f807b268e9c61ef795c45d59cff3f5be51d2583aa46a7c00a9cf09b"} Oct 30 06:41:41.518706 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:41.518676 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-plugins-e2e-extmod-a-2d2923: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2d2923" not found Oct 30 06:41:41.518891 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:41.518752 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume podName:1665a272-058b-4a3b-8058-9c92c47476e1 nodeName:}" failed. No retries permitted until 2026-10-30 06:41:42.018732207 +0000 UTC m=+2916.014689198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2d2923" not found Oct 30 06:41:41.619909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.619872 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:41:41.721792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.721760 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-provider-credentials\") pod \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " Oct 30 06:41:41.722004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.721826 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-kube-api-access-4h5fs\" (UniqueName: \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-kube-api-access-4h5fs\") pod \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " Oct 30 06:41:41.722004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.721855 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-plugins-config-volume\") pod \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " Oct 30 06:41:41.722231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.722206 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-routing-overlay-volume\") pod \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\" (UID: \"9aaf92ff-2a28-4d87-97e9-06e555adb83d\") " Oct 30 06:41:41.722445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.722422 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-plugins-config-volume" pod "9aaf92ff-2a28-4d87-97e9-06e555adb83d" (UID: "9aaf92ff-2a28-4d87-97e9-06e555adb83d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:41.722640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.722609 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-routing-overlay-volume" pod "9aaf92ff-2a28-4d87-97e9-06e555adb83d" (UID: "9aaf92ff-2a28-4d87-97e9-06e555adb83d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:41.723056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.723035 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:41.723107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.723077 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aaf92ff-2a28-4d87-97e9-06e555adb83d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:41.724656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.724630 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-kube-api-access-4h5fs" pod "9aaf92ff-2a28-4d87-97e9-06e555adb83d" (UID: "9aaf92ff-2a28-4d87-97e9-06e555adb83d"). InnerVolumeSpecName "kube-api-access-4h5fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:41.725135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.725111 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-provider-credentials" pod "9aaf92ff-2a28-4d87-97e9-06e555adb83d" (UID: "9aaf92ff-2a28-4d87-97e9-06e555adb83d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:41.824831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.824794 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:41.824831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:41.824825 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h5fs\" (UniqueName: \"kubernetes.io/projected/9aaf92ff-2a28-4d87-97e9-06e555adb83d-kube-api-access-4h5fs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:42.028491 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:42.028455 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-plugins-e2e-extmod-a-2d2923: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2d2923" not found Oct 30 06:41:42.028691 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:42.028543 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume podName:1665a272-058b-4a3b-8058-9c92c47476e1 nodeName:}" failed. No retries permitted until 2026-10-30 06:41:43.028524854 +0000 UTC m=+2917.024481864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2d2923" not found Oct 30 06:41:42.028691 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:42.028622 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d2923/agc-cred-c32bb415fa0e: secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:42.028691 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:42.028645 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg: secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:42.028691 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:42.028693 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials podName:1665a272-058b-4a3b-8058-9c92c47476e1 nodeName:}" failed. No retries permitted until 2026-10-30 06:41:44.028680458 +0000 UTC m=+2918.024637456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1") : secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:42.475612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:42.475574 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" Oct 30 06:41:42.475612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:42.475592 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h" event={"ID":"9aaf92ff-2a28-4d87-97e9-06e555adb83d","Type":"ContainerDied","Data":"ff9f6f2d12a359c706ca9d14c36b97a94d5ca44e38b7e1b82a3d7d2d556f6bb2"} Oct 30 06:41:42.476234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:42.475668 5001 scope.go:122] "RemoveContainer" containerID="d07b68051f807b268e9c61ef795c45d59cff3f5be51d2583aa46a7c00a9cf09b" Oct 30 06:41:42.580405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:42.580368 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h"] Oct 30 06:41:42.585600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:42.585573 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-69f585c47c-wkp5h"] Oct 30 06:41:43.064335 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:43.057900 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-plugins-e2e-extmod-a-2d2923: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2d2923" not found Oct 30 06:41:43.064335 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:43.057987 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume podName:1665a272-058b-4a3b-8058-9c92c47476e1 nodeName:}" failed. No retries permitted until 2026-10-30 06:41:45.057966357 +0000 UTC m=+2919.053923351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2d2923" not found Oct 30 06:41:44.118040 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:44.118007 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d2923/agc-cred-c32bb415fa0e: secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:44.118040 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:44.118038 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg: secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:44.118572 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:44.118106 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials podName:1665a272-058b-4a3b-8058-9c92c47476e1 nodeName:}" failed. No retries permitted until 2026-10-30 06:41:48.118088388 +0000 UTC m=+2922.114045395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1") : secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:44.120437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.120412 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aaf92ff-2a28-4d87-97e9-06e555adb83d" path="/var/lib/kubelet/pods/9aaf92ff-2a28-4d87-97e9-06e555adb83d/volumes" Oct 30 06:41:44.415688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.415605 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd"] Oct 30 06:41:44.416504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.416488 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9aaf92ff-2a28-4d87-97e9-06e555adb83d" containerName="payload-processing" Oct 30 06:41:44.416553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.416505 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aaf92ff-2a28-4d87-97e9-06e555adb83d" containerName="payload-processing" Oct 30 06:41:44.416634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.416625 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aaf92ff-2a28-4d87-97e9-06e555adb83d" containerName="payload-processing" Oct 30 06:41:44.421969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.421950 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.441424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.441392 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd"] Oct 30 06:41:44.447321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.447280 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8b3017c11d13\"" Oct 30 06:41:44.524814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.524777 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-routing-overlay-volume\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.525018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.524998 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-provider-credentials\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.525204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.525182 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-plugins-config-volume\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.525266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.525217 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhtjt\" (UniqueName: \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-kube-api-access-nhtjt\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.626944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.626909 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-provider-credentials\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.627114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.627089 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-plugins-config-volume\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.627187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.627121 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtjt\" (UniqueName: \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-kube-api-access-nhtjt\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.627238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.627221 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-routing-overlay-volume\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.627906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.627831 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-plugins-config-volume\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.627906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.627859 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-routing-overlay-volume\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.629868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.629846 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-provider-credentials\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.679918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.679829 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhtjt\" (UniqueName: \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-kube-api-access-nhtjt\") pod \"payload-processing-external-model-5b5448f9b9-k56kd\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:44.739467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:44.739433 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:45.064866 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:41:45.064825 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95870ca_ad71_47e2_bc5b_7d23c88ddd64.slice/crio-3d34dc1fdde6b06d34e72ce3becb829f040aba96421e770884fa9a33cdea73d1 WatchSource:0}: Error finding container 3d34dc1fdde6b06d34e72ce3becb829f040aba96421e770884fa9a33cdea73d1: Status 404 returned error can't find the container with id 3d34dc1fdde6b06d34e72ce3becb829f040aba96421e770884fa9a33cdea73d1 Oct 30 06:41:45.067771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:45.067732 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd"] Oct 30 06:41:45.140022 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:45.139997 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-plugins-e2e-extmod-a-2d2923: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2d2923" not found Oct 30 06:41:45.140478 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:45.140079 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume podName:1665a272-058b-4a3b-8058-9c92c47476e1 nodeName:}" failed. No retries permitted until 2026-10-30 06:41:49.140060186 +0000 UTC m=+2923.136017173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2d2923" not found Oct 30 06:41:45.616200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:45.616160 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" event={"ID":"e95870ca-ad71-47e2-bc5b-7d23c88ddd64","Type":"ContainerStarted","Data":"d5477d5d47f535fc0a33fabbf27d88a0a7a888d22f997abfa3356079fd733511"} Oct 30 06:41:45.616469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:45.616242 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" event={"ID":"e95870ca-ad71-47e2-bc5b-7d23c88ddd64","Type":"ContainerStarted","Data":"3d34dc1fdde6b06d34e72ce3becb829f040aba96421e770884fa9a33cdea73d1"} Oct 30 06:41:45.616469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:45.616275 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:45.662822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:45.662776 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" podStartSLOduration=1.662761921 podStartE2EDuration="1.662761921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:41:45.658521285 +0000 UTC m=+2919.654478315" watchObservedRunningTime="2026-10-30 06:41:45.662761921 +0000 UTC m=+2919.658718931" Oct 30 06:41:47.498397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:47.481752 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" podUID="1665a272-058b-4a3b-8058-9c92c47476e1" containerName="payload-processing" containerID="cri-o://1aecdd0de1dae5cbbea1837f3df9b7c269c9c425d6a4a7eab1af1f6fafa2d296" gracePeriod=25 Oct 30 06:41:48.281380 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:48.280653 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d2923/agc-cred-c32bb415fa0e: secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:48.281380 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:48.280680 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg: secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:48.281380 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:41:48.280736 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials podName:1665a272-058b-4a3b-8058-9c92c47476e1 nodeName:}" failed. No retries permitted until 2026-10-30 06:41:56.280719454 +0000 UTC m=+2930.276676442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1") : secret "agc-cred-c32bb415fa0e" not found Oct 30 06:41:48.390329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.389389 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd"] Oct 30 06:41:48.768074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.760070 5001 generic.go:356] "Generic (PLEG): container finished" podID="1665a272-058b-4a3b-8058-9c92c47476e1" containerID="1aecdd0de1dae5cbbea1837f3df9b7c269c9c425d6a4a7eab1af1f6fafa2d296" exitCode=0 Oct 30 06:41:48.768074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.760350 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" event={"ID":"1665a272-058b-4a3b-8058-9c92c47476e1","Type":"ContainerDied","Data":"1aecdd0de1dae5cbbea1837f3df9b7c269c9c425d6a4a7eab1af1f6fafa2d296"} Oct 30 06:41:48.791002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.790956 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:48.877546 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.877508 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-routing-overlay-volume\") pod \"1665a272-058b-4a3b-8058-9c92c47476e1\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " Oct 30 06:41:48.877777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.877719 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume\") pod \"1665a272-058b-4a3b-8058-9c92c47476e1\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " Oct 30 06:41:48.877871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.877838 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials\") pod \"1665a272-058b-4a3b-8058-9c92c47476e1\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " Oct 30 06:41:48.877929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.877867 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-kube-api-access-qf4rs\" (UniqueName: \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-kube-api-access-qf4rs\") pod \"1665a272-058b-4a3b-8058-9c92c47476e1\" (UID: \"1665a272-058b-4a3b-8058-9c92c47476e1\") " Oct 30 06:41:48.878188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.878155 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume" pod "1665a272-058b-4a3b-8058-9c92c47476e1" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:48.878742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.878713 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:48.880198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.880171 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-routing-overlay-volume" pod "1665a272-058b-4a3b-8058-9c92c47476e1" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:48.881197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.881171 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-kube-api-access-qf4rs" pod "1665a272-058b-4a3b-8058-9c92c47476e1" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1"). InnerVolumeSpecName "kube-api-access-qf4rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:48.881294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.881212 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials" pod "1665a272-058b-4a3b-8058-9c92c47476e1" (UID: "1665a272-058b-4a3b-8058-9c92c47476e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:48.980038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.979995 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1665a272-058b-4a3b-8058-9c92c47476e1-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:48.980038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.980026 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:48.980038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:48.980036 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf4rs\" (UniqueName: \"kubernetes.io/projected/1665a272-058b-4a3b-8058-9c92c47476e1-kube-api-access-qf4rs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:49.775497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:49.775442 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" event={"ID":"1665a272-058b-4a3b-8058-9c92c47476e1","Type":"ContainerDied","Data":"f93ddb2a112a843ff6a789721af1adeab160f78ff46dae0f2c9c04fd8fe0ab9a"} Oct 30 06:41:49.776211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:49.776179 5001 scope.go:122] "RemoveContainer" containerID="1aecdd0de1dae5cbbea1837f3df9b7c269c9c425d6a4a7eab1af1f6fafa2d296" Oct 30 06:41:49.776654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:49.776637 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg" Oct 30 06:41:49.875420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:49.875384 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg"] Oct 30 06:41:49.926846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:49.926803 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-7c6fgqvjg"] Oct 30 06:41:50.128392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:50.128361 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1665a272-058b-4a3b-8058-9c92c47476e1" path="/var/lib/kubelet/pods/1665a272-058b-4a3b-8058-9c92c47476e1/volumes" Oct 30 06:41:53.393331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.392381 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" podUID="e95870ca-ad71-47e2-bc5b-7d23c88ddd64" containerName="payload-processing" containerID="cri-o://d5477d5d47f535fc0a33fabbf27d88a0a7a888d22f997abfa3356079fd733511" gracePeriod=25 Oct 30 06:41:53.922126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.921256 5001 generic.go:356] "Generic (PLEG): container finished" podID="e95870ca-ad71-47e2-bc5b-7d23c88ddd64" containerID="d5477d5d47f535fc0a33fabbf27d88a0a7a888d22f997abfa3356079fd733511" exitCode=0 Oct 30 06:41:53.922126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.921951 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" event={"ID":"e95870ca-ad71-47e2-bc5b-7d23c88ddd64","Type":"ContainerDied","Data":"d5477d5d47f535fc0a33fabbf27d88a0a7a888d22f997abfa3356079fd733511"} Oct 30 06:41:53.922126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.922053 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" event={"ID":"e95870ca-ad71-47e2-bc5b-7d23c88ddd64","Type":"ContainerDied","Data":"3d34dc1fdde6b06d34e72ce3becb829f040aba96421e770884fa9a33cdea73d1"} Oct 30 06:41:53.922126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.922083 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d34dc1fdde6b06d34e72ce3becb829f040aba96421e770884fa9a33cdea73d1" Oct 30 06:41:53.931403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.931377 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:53.975567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.975518 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-plugins-config-volume\") pod \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " Oct 30 06:41:53.975826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.975587 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-routing-overlay-volume\") pod \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " Oct 30 06:41:53.976217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.976188 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-routing-overlay-volume" pod "e95870ca-ad71-47e2-bc5b-7d23c88ddd64" (UID: "e95870ca-ad71-47e2-bc5b-7d23c88ddd64"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:53.976217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.976205 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-plugins-config-volume" pod "e95870ca-ad71-47e2-bc5b-7d23c88ddd64" (UID: "e95870ca-ad71-47e2-bc5b-7d23c88ddd64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:53.976432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.976272 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-kube-api-access-nhtjt\" (UniqueName: \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-kube-api-access-nhtjt\") pod \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " Oct 30 06:41:53.976432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.976406 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-provider-credentials\") pod \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\" (UID: \"e95870ca-ad71-47e2-bc5b-7d23c88ddd64\") " Oct 30 06:41:53.977170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.977049 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:53.977170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.977072 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:53.979634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.979607 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-provider-credentials" pod "e95870ca-ad71-47e2-bc5b-7d23c88ddd64" (UID: "e95870ca-ad71-47e2-bc5b-7d23c88ddd64"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:53.981505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:53.981478 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-kube-api-access-nhtjt" pod "e95870ca-ad71-47e2-bc5b-7d23c88ddd64" (UID: "e95870ca-ad71-47e2-bc5b-7d23c88ddd64"). InnerVolumeSpecName "kube-api-access-nhtjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:54.078754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:54.078714 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhtjt\" (UniqueName: \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-kube-api-access-nhtjt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:54.078754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:54.078753 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e95870ca-ad71-47e2-bc5b-7d23c88ddd64-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:41:55.098338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:55.096294 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd" Oct 30 06:41:55.259327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:55.245501 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd"] Oct 30 06:41:55.268819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:55.268655 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-k56kd"] Oct 30 06:41:56.067877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.067055 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf"] Oct 30 06:41:56.068495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.068468 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1665a272-058b-4a3b-8058-9c92c47476e1" containerName="payload-processing" Oct 30 06:41:56.068689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.068670 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1665a272-058b-4a3b-8058-9c92c47476e1" containerName="payload-processing" Oct 30 06:41:56.068902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.068880 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e95870ca-ad71-47e2-bc5b-7d23c88ddd64" containerName="payload-processing" Oct 30 06:41:56.068902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.068900 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95870ca-ad71-47e2-bc5b-7d23c88ddd64" containerName="payload-processing" Oct 30 06:41:56.069146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.069130 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1665a272-058b-4a3b-8058-9c92c47476e1" containerName="payload-processing" Oct 30 06:41:56.069219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.069162 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95870ca-ad71-47e2-bc5b-7d23c88ddd64" containerName="payload-processing" Oct 30 06:41:56.073743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.073721 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.092929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.092899 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf"] Oct 30 06:41:56.104623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.104593 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b5bb1149aea2\"" Oct 30 06:41:56.105041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.105018 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8b3017c11d13\"" Oct 30 06:41:56.121592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.121562 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95870ca-ad71-47e2-bc5b-7d23c88ddd64" path="/var/lib/kubelet/pods/e95870ca-ad71-47e2-bc5b-7d23c88ddd64/volumes" Oct 30 06:41:56.170927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.170880 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-routing-overlay-volume\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.171149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.171065 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnz4x\" (UniqueName: \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-kube-api-access-gnz4x\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.171222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.171148 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-provider-credentials\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.171289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.171270 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-plugins-config-volume\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.272991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.272951 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-plugins-config-volume\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.273213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.273082 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-routing-overlay-volume\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.273289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.273243 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz4x\" (UniqueName: \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-kube-api-access-gnz4x\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.273407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.273388 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-provider-credentials\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.273876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.273847 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-routing-overlay-volume\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.274021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.273933 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-plugins-config-volume\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.276443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.276415 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-provider-credentials\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.298260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.298225 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnz4x\" (UniqueName: \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-kube-api-access-gnz4x\") pod \"payload-processing-external-model-6969648bc7-mwfbf\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.392663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.392624 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:56.709891 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:41:56.709850 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31a2396_5256_44fe_9c46_5e19d227b34a.slice/crio-3309847251f817b9e33b5ee77228b5e1da629fd3d819cd86c0ffba967a7d13c6 WatchSource:0}: Error finding container 3309847251f817b9e33b5ee77228b5e1da629fd3d819cd86c0ffba967a7d13c6: Status 404 returned error can't find the container with id 3309847251f817b9e33b5ee77228b5e1da629fd3d819cd86c0ffba967a7d13c6 Oct 30 06:41:56.710034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:56.710011 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf"] Oct 30 06:41:57.124929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:57.124897 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" event={"ID":"d31a2396-5256-44fe-9c46-5e19d227b34a","Type":"ContainerStarted","Data":"48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e"} Oct 30 06:41:57.125341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:57.124950 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" event={"ID":"d31a2396-5256-44fe-9c46-5e19d227b34a","Type":"ContainerStarted","Data":"3309847251f817b9e33b5ee77228b5e1da629fd3d819cd86c0ffba967a7d13c6"} Oct 30 06:41:57.125341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:57.124972 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:41:57.176329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:57.176094 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" podStartSLOduration=1.176073428 podStartE2EDuration="1.176073428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:41:57.168096456 +0000 UTC m=+2931.164053467" watchObservedRunningTime="2026-10-30 06:41:57.176073428 +0000 UTC m=+2931.172030440" Oct 30 06:41:58.713009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:41:58.712974 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf"] Oct 30 06:42:01.383325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.383262 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-58946db44c-tdkhz"] Oct 30 06:42:01.388749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.388724 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.408298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.408269 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58946db44c-tdkhz"] Oct 30 06:42:01.414116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.414088 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c537891062d6\"" Oct 30 06:42:01.511750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.511698 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p45hp\" (UniqueName: \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-kube-api-access-p45hp\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.515328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.512156 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-provider-credentials\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.515699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.515671 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-routing-overlay-volume\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.521340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.516002 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-plugins-config-volume\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.629551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.629505 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-provider-credentials\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.629890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.629870 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-routing-overlay-volume\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.630142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.630123 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-plugins-config-volume\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.630376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.630357 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p45hp\" (UniqueName: \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-kube-api-access-p45hp\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.632171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.632144 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-routing-overlay-volume\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.633192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.633088 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-plugins-config-volume\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.672462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.672380 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-provider-credentials\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.726248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.726196 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p45hp\" (UniqueName: \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-kube-api-access-p45hp\") pod \"payload-processing-external-model-58946db44c-tdkhz\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:01.771999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.771940 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p"] Oct 30 06:42:01.786779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.786747 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.863148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.863050 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.863437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.863417 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.863673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.863657 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtphh\" (UniqueName: \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-kube-api-access-vtphh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.863886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.863872 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.864088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.864074 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.864293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.864279 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.864483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.864469 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.864669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.864655 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.864967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.864949 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.876550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.876445 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p"] Oct 30 06:42:01.990413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.989734 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.990413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.989825 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.990413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.989888 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.990413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.989944 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtphh\" (UniqueName: \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-kube-api-access-vtphh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.990413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.990000 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.990413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.990040 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.990413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.990099 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.990413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.990134 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.990413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.990168 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.991052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.990522 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.991052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.990597 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.991052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.990677 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.991246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.991141 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.991246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.991177 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.994371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.993546 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:01.995042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:01.994995 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:02.008330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:02.006407 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:02.024072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:02.024004 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtphh\" (UniqueName: \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-kube-api-access-vtphh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:02.025911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:02.025443 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4fh4p\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:02.152632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:02.152590 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:02.356810 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:42:02.355502 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fbee68_0311_4e52_bc0b_a64bf7b161ca.slice/crio-52823f63ed668f4051d92753615e5020d6108fd37a1bc69aa6ea64667789a08d WatchSource:0}: Error finding container 52823f63ed668f4051d92753615e5020d6108fd37a1bc69aa6ea64667789a08d: Status 404 returned error can't find the container with id 52823f63ed668f4051d92753615e5020d6108fd37a1bc69aa6ea64667789a08d Oct 30 06:42:02.358600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:02.358573 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58946db44c-tdkhz"] Oct 30 06:42:02.500806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:02.500775 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p"] Oct 30 06:42:02.501203 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:42:02.500939 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc4b90d_d07f_4f35_91ea_41dca25a573d.slice/crio-75daeba016f0387c788cb0d1135c849f3bdd7f55159d73bc6508014551d4d89f WatchSource:0}: Error finding container 75daeba016f0387c788cb0d1135c849f3bdd7f55159d73bc6508014551d4d89f: Status 404 returned error can't find the container with id 75daeba016f0387c788cb0d1135c849f3bdd7f55159d73bc6508014551d4d89f Oct 30 06:42:02.504095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:02.504055 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:42:02.504179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:02.504127 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:42:02.504179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:02.504161 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:42:03.390577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:03.390536 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" event={"ID":"15fbee68-0311-4e52-bc0b-a64bf7b161ca","Type":"ContainerStarted","Data":"2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1"} Oct 30 06:42:03.390794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:03.390606 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" event={"ID":"15fbee68-0311-4e52-bc0b-a64bf7b161ca","Type":"ContainerStarted","Data":"52823f63ed668f4051d92753615e5020d6108fd37a1bc69aa6ea64667789a08d"} Oct 30 06:42:03.390794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:03.390633 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:03.392751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:03.392728 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" event={"ID":"1fc4b90d-d07f-4f35-91ea-41dca25a573d","Type":"ContainerStarted","Data":"372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3"} Oct 30 06:42:03.392873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:03.392771 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" event={"ID":"1fc4b90d-d07f-4f35-91ea-41dca25a573d","Type":"ContainerStarted","Data":"75daeba016f0387c788cb0d1135c849f3bdd7f55159d73bc6508014551d4d89f"} Oct 30 06:42:03.546672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:03.546618 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podStartSLOduration=2.54660329 podStartE2EDuration="2.54660329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:42:03.542852242 +0000 UTC m=+2937.538809256" watchObservedRunningTime="2026-10-30 06:42:03.54660329 +0000 UTC m=+2937.542560380" Oct 30 06:42:03.547113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:03.546893 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" podStartSLOduration=2.546887011 podStartE2EDuration="2.546887011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:42:03.482778389 +0000 UTC m=+2937.478735399" watchObservedRunningTime="2026-10-30 06:42:03.546887011 +0000 UTC m=+2937.542844021" Oct 30 06:42:04.176167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:04.175388 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:04.178906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:04.178875 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4fh4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Oct 30 06:42:04.179125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:04.179099 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Oct 30 06:42:04.341327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:04.340372 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" podUID="d31a2396-5256-44fe-9c46-5e19d227b34a" containerName="payload-processing" containerID="cri-o://48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e" gracePeriod=25 Oct 30 06:42:05.150892 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.150864 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:42:05.162336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.158707 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4fh4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Oct 30 06:42:05.162336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.158764 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Oct 30 06:42:05.234010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.233979 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-kube-api-access-gnz4x\" (UniqueName: \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-kube-api-access-gnz4x\") pod \"d31a2396-5256-44fe-9c46-5e19d227b34a\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " Oct 30 06:42:05.234187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.234034 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-provider-credentials\") pod \"d31a2396-5256-44fe-9c46-5e19d227b34a\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " Oct 30 06:42:05.234187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.234152 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-plugins-config-volume\") pod \"d31a2396-5256-44fe-9c46-5e19d227b34a\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " Oct 30 06:42:05.234406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.234233 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-routing-overlay-volume\") pod \"d31a2396-5256-44fe-9c46-5e19d227b34a\" (UID: \"d31a2396-5256-44fe-9c46-5e19d227b34a\") " Oct 30 06:42:05.234661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.234633 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-plugins-config-volume" pod "d31a2396-5256-44fe-9c46-5e19d227b34a" (UID: "d31a2396-5256-44fe-9c46-5e19d227b34a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:05.235522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.234865 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-routing-overlay-volume" pod "d31a2396-5256-44fe-9c46-5e19d227b34a" (UID: "d31a2396-5256-44fe-9c46-5e19d227b34a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:05.235522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.235398 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:05.235522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.235424 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31a2396-5256-44fe-9c46-5e19d227b34a-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:05.236798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.236779 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-provider-credentials" pod "d31a2396-5256-44fe-9c46-5e19d227b34a" (UID: "d31a2396-5256-44fe-9c46-5e19d227b34a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:05.237355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.237327 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-kube-api-access-gnz4x" pod "d31a2396-5256-44fe-9c46-5e19d227b34a" (UID: "d31a2396-5256-44fe-9c46-5e19d227b34a"). InnerVolumeSpecName "kube-api-access-gnz4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:05.337618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.337576 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnz4x\" (UniqueName: \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-kube-api-access-gnz4x\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:05.337618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.337620 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d31a2396-5256-44fe-9c46-5e19d227b34a-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:05.580826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.580737 5001 generic.go:356] "Generic (PLEG): container finished" podID="d31a2396-5256-44fe-9c46-5e19d227b34a" containerID="48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e" exitCode=0 Oct 30 06:42:05.597330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.581097 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" Oct 30 06:42:05.601683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.581121 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" event={"ID":"d31a2396-5256-44fe-9c46-5e19d227b34a","Type":"ContainerDied","Data":"48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e"} Oct 30 06:42:05.618212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.618178 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf" event={"ID":"d31a2396-5256-44fe-9c46-5e19d227b34a","Type":"ContainerDied","Data":"3309847251f817b9e33b5ee77228b5e1da629fd3d819cd86c0ffba967a7d13c6"} Oct 30 06:42:05.618531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.618467 5001 scope.go:122] "RemoveContainer" containerID="48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e" Oct 30 06:42:05.758333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.752139 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv"] Oct 30 06:42:05.758333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.753579 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d31a2396-5256-44fe-9c46-5e19d227b34a" containerName="payload-processing" Oct 30 06:42:05.758333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.753602 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31a2396-5256-44fe-9c46-5e19d227b34a" containerName="payload-processing" Oct 30 06:42:05.758333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:05.753885 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31a2396-5256-44fe-9c46-5e19d227b34a" containerName="payload-processing" Oct 30 06:42:06.038331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.037643 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv"] Oct 30 06:42:06.038331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.037698 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf"] Oct 30 06:42:06.062580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.038781 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.082331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.068787 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-plugins-e2e-extmod-a-88acce\"" Oct 30 06:42:06.099937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.095612 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"agc-cred-39512e62d644\"" Oct 30 06:42:06.099937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.095854 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-e2e--4e48ea35-dockercfg-v22pn\"" Oct 30 06:42:06.099937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.096735 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"openshift-service-ca.crt\"" Oct 30 06:42:06.099937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.096913 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"kube-root-ca.crt\"" Oct 30 06:42:06.099937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.097154 5001 scope.go:122] "RemoveContainer" containerID="48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e" Oct 30 06:42:06.125331 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:42:06.121926 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e\": container with ID starting with 48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e not found: ID does not exist" containerID="48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e" Oct 30 06:42:06.125331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.121976 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e"} err="failed to get container status \"48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e\": rpc error: code = NotFound desc = could not find container \"48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e\": container with ID starting with 48647d421fdc4c7b3bea93857e0c4ff7db2c2e6416f1da3da87b232f19a7349e not found: ID does not exist" Oct 30 06:42:06.183176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.175753 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"routing-overlay\"" Oct 30 06:42:06.298380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.297840 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.298380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.297969 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6647\" (UniqueName: \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-kube-api-access-h6647\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.298380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.298004 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.298380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.298051 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.408334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.400972 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.408334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.401090 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6647\" (UniqueName: \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-kube-api-access-h6647\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.408334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.401126 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.408334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.401170 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.410840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.409725 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4fh4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Oct 30 06:42:06.410840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.409777 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Oct 30 06:42:06.417472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.416939 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"routing-overlay\"" Oct 30 06:42:06.417864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.417799 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-mwfbf"] Oct 30 06:42:06.419900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.419860 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"agc-cred-39512e62d644\"" Oct 30 06:42:06.420038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.419869 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-plugins-e2e-extmod-a-88acce\"" Oct 30 06:42:06.422628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.422606 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.426441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.426425 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.428912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.428884 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.441277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.441252 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"kube-root-ca.crt\"" Oct 30 06:42:06.469332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.468597 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"openshift-service-ca.crt\"" Oct 30 06:42:06.481594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.481567 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6647\" (UniqueName: \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-kube-api-access-h6647\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:06.759334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.757990 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-e2e--4e48ea35-dockercfg-v22pn\"" Oct 30 06:42:06.785345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:06.775673 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:07.166329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:07.165791 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4fh4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Oct 30 06:42:07.166329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:07.165853 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Oct 30 06:42:07.638136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:07.638105 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv"] Oct 30 06:42:08.174331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:08.167264 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4fh4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Oct 30 06:42:08.174331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:08.167361 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Oct 30 06:42:08.187548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:08.187506 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31a2396-5256-44fe-9c46-5e19d227b34a" path="/var/lib/kubelet/pods/d31a2396-5256-44fe-9c46-5e19d227b34a/volumes" Oct 30 06:42:08.293783 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:42:08.293737 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:42:08.630150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:08.630111 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" event={"ID":"3186c702-d417-4137-93ad-81abbb463f5e","Type":"ContainerStarted","Data":"f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944"} Oct 30 06:42:08.630353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:08.630174 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" event={"ID":"3186c702-d417-4137-93ad-81abbb463f5e","Type":"ContainerStarted","Data":"9453b08a48cecafb22657727eb62093bf30eacbcd7594ffbf4126520078486e2"} Oct 30 06:42:08.630353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:08.630203 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:08.695396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:08.695334 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" podStartSLOduration=3.695296388 podStartE2EDuration="3.695296388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:42:08.694584359 +0000 UTC m=+2942.690541369" watchObservedRunningTime="2026-10-30 06:42:08.695296388 +0000 UTC m=+2942.691253401" Oct 30 06:42:09.156342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:09.155697 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4fh4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Oct 30 06:42:09.156342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:09.155761 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Oct 30 06:42:10.153919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:10.153881 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4fh4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Oct 30 06:42:10.154481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:10.153936 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Oct 30 06:42:10.920601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:10.920568 5001 scope.go:122] "RemoveContainer" containerID="1b3f4bc64e013a698e90a35c7f59a84e4019ce53d4fee8b28f227596cf19be8d" Oct 30 06:42:11.153884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:11.153844 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4fh4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Oct 30 06:42:11.154049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:11.153912 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Oct 30 06:42:13.286549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:13.286491 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4fh4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:42:13.287081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:13.286591 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:42:13.526685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:13.526650 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:13.527240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:13.527217 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:13.528081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:13.528060 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:42:14.475561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:14.475527 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:42:14.580934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:14.580900 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54cb654cf-98j9x"] Oct 30 06:42:19.000004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:18.999958 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:42:19.582233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:19.582184 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" podUID="4d01dd55-de02-4f0c-83d6-303271866adf" containerName="payload-processing" containerID="cri-o://88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c" gracePeriod=25 Oct 30 06:42:19.642784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:19.642754 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:20.279079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.279038 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:42:20.293867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.293827 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-plugins-config-volume\") pod \"4d01dd55-de02-4f0c-83d6-303271866adf\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " Oct 30 06:42:20.294213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.294193 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-provider-credentials\") pod \"4d01dd55-de02-4f0c-83d6-303271866adf\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " Oct 30 06:42:20.294279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.294234 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-routing-overlay-volume\") pod \"4d01dd55-de02-4f0c-83d6-303271866adf\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " Oct 30 06:42:20.294359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.294325 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-kube-api-access-rs9jq\" (UniqueName: \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-kube-api-access-rs9jq\") pod \"4d01dd55-de02-4f0c-83d6-303271866adf\" (UID: \"4d01dd55-de02-4f0c-83d6-303271866adf\") " Oct 30 06:42:20.294359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.294335 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-plugins-config-volume" pod "4d01dd55-de02-4f0c-83d6-303271866adf" (UID: "4d01dd55-de02-4f0c-83d6-303271866adf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:20.294944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.294913 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-routing-overlay-volume" pod "4d01dd55-de02-4f0c-83d6-303271866adf" (UID: "4d01dd55-de02-4f0c-83d6-303271866adf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:20.295178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.295140 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:20.295178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.295167 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d01dd55-de02-4f0c-83d6-303271866adf-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:20.297076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.297047 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-kube-api-access-rs9jq" pod "4d01dd55-de02-4f0c-83d6-303271866adf" (UID: "4d01dd55-de02-4f0c-83d6-303271866adf"). InnerVolumeSpecName "kube-api-access-rs9jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:20.297209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.297085 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-provider-credentials" pod "4d01dd55-de02-4f0c-83d6-303271866adf" (UID: "4d01dd55-de02-4f0c-83d6-303271866adf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:20.396776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.396728 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:20.396776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.396771 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9jq\" (UniqueName: \"kubernetes.io/projected/4d01dd55-de02-4f0c-83d6-303271866adf-kube-api-access-rs9jq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:20.411869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.411829 5001 generic.go:356] "Generic (PLEG): container finished" podID="4d01dd55-de02-4f0c-83d6-303271866adf" containerID="88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c" exitCode=0 Oct 30 06:42:20.412090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.411902 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" Oct 30 06:42:20.412090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.411923 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" event={"ID":"4d01dd55-de02-4f0c-83d6-303271866adf","Type":"ContainerDied","Data":"88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c"} Oct 30 06:42:20.412090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.412028 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54cb654cf-98j9x" event={"ID":"4d01dd55-de02-4f0c-83d6-303271866adf","Type":"ContainerDied","Data":"f8c4e5452557cc4190342bb1368db140b4e7d7537406149713ae4d325ee53bd0"} Oct 30 06:42:20.412090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.412062 5001 scope.go:122] "RemoveContainer" containerID="88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c" Oct 30 06:42:20.425416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.425393 5001 scope.go:122] "RemoveContainer" containerID="88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c" Oct 30 06:42:20.425771 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:42:20.425748 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c\": container with ID starting with 88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c not found: ID does not exist" containerID="88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c" Oct 30 06:42:20.425825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.425785 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c"} err="failed to get container status \"88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c\": rpc error: code = NotFound desc = could not find container \"88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c\": container with ID starting with 88b62ca6a57636d10bb84969c734120bc8e49e6774fe12c98d79f1335a7a409c not found: ID does not exist" Oct 30 06:42:20.470771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.470733 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54cb654cf-98j9x"] Oct 30 06:42:20.476245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:20.476216 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-54cb654cf-98j9x"] Oct 30 06:42:22.141018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:22.138726 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d01dd55-de02-4f0c-83d6-303271866adf" path="/var/lib/kubelet/pods/4d01dd55-de02-4f0c-83d6-303271866adf/volumes" Oct 30 06:42:26.758279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:26.758244 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv"] Oct 30 06:42:31.759576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:31.759523 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" podUID="3186c702-d417-4137-93ad-81abbb463f5e" containerName="payload-processing" containerID="cri-o://f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944" gracePeriod=25 Oct 30 06:42:32.060514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.060487 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:32.152060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.152016 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-plugins-config-volume\") pod \"3186c702-d417-4137-93ad-81abbb463f5e\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " Oct 30 06:42:32.152299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.152280 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-routing-overlay-volume\") pod \"3186c702-d417-4137-93ad-81abbb463f5e\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " Oct 30 06:42:32.152459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.152377 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-provider-credentials\") pod \"3186c702-d417-4137-93ad-81abbb463f5e\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " Oct 30 06:42:32.152532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.152490 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-plugins-config-volume" pod "3186c702-d417-4137-93ad-81abbb463f5e" (UID: "3186c702-d417-4137-93ad-81abbb463f5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:32.152601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.152550 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-kube-api-access-h6647\" (UniqueName: \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-kube-api-access-h6647\") pod \"3186c702-d417-4137-93ad-81abbb463f5e\" (UID: \"3186c702-d417-4137-93ad-81abbb463f5e\") " Oct 30 06:42:32.152959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.152932 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-routing-overlay-volume" pod "3186c702-d417-4137-93ad-81abbb463f5e" (UID: "3186c702-d417-4137-93ad-81abbb463f5e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:32.153471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.153446 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:32.153672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.153636 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3186c702-d417-4137-93ad-81abbb463f5e-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:32.157597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.157569 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-provider-credentials" pod "3186c702-d417-4137-93ad-81abbb463f5e" (UID: "3186c702-d417-4137-93ad-81abbb463f5e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:32.157975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.157952 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-kube-api-access-h6647" pod "3186c702-d417-4137-93ad-81abbb463f5e" (UID: "3186c702-d417-4137-93ad-81abbb463f5e"). InnerVolumeSpecName "kube-api-access-h6647". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:32.255002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.254964 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:32.255002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.254995 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6647\" (UniqueName: \"kubernetes.io/projected/3186c702-d417-4137-93ad-81abbb463f5e-kube-api-access-h6647\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:42:32.642484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.642446 5001 generic.go:356] "Generic (PLEG): container finished" podID="3186c702-d417-4137-93ad-81abbb463f5e" containerID="f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944" exitCode=0 Oct 30 06:42:32.642865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.642839 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" Oct 30 06:42:32.643135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.643104 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" event={"ID":"3186c702-d417-4137-93ad-81abbb463f5e","Type":"ContainerDied","Data":"f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944"} Oct 30 06:42:32.643251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.643212 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv" event={"ID":"3186c702-d417-4137-93ad-81abbb463f5e","Type":"ContainerDied","Data":"9453b08a48cecafb22657727eb62093bf30eacbcd7594ffbf4126520078486e2"} Oct 30 06:42:32.643339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.643259 5001 scope.go:122] "RemoveContainer" containerID="f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944" Oct 30 06:42:32.661434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.661414 5001 scope.go:122] "RemoveContainer" containerID="f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944" Oct 30 06:42:32.661980 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:42:32.661955 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944\": container with ID starting with f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944 not found: ID does not exist" containerID="f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944" Oct 30 06:42:32.662065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.661991 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944"} err="failed to get container status \"f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944\": rpc error: code = NotFound desc = could not find container \"f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944\": container with ID starting with f691c0bb402f3bc18740fee7598f5b1fb96081043eaa04e16fdc69ca4ee49944 not found: ID does not exist" Oct 30 06:42:32.701547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.701512 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv"] Oct 30 06:42:32.713914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:32.713882 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6hqfsv"] Oct 30 06:42:34.119942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:34.119910 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3186c702-d417-4137-93ad-81abbb463f5e" path="/var/lib/kubelet/pods/3186c702-d417-4137-93ad-81abbb463f5e/volumes" Oct 30 06:42:38.274213 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:42:38.274173 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:42:40.041127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:40.041091 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv"] Oct 30 06:42:40.052801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:40.052768 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9063d552-vmjwv"] Oct 30 06:42:40.125019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:40.124111 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b11efe0-86be-4e43-994c-06ed54464099" path="/var/lib/kubelet/pods/6b11efe0-86be-4e43-994c-06ed54464099/volumes" Oct 30 06:42:57.211394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:57.211360 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm"] Oct 30 06:42:57.211972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:57.211608 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" containerID="cri-o://9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5" gracePeriod=30 Oct 30 06:42:59.884209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:42:59.884174 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:43:02.773749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.769806 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:43:02.914390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.913320 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-kube-api-access-kmdvv\" (UniqueName: \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-kube-api-access-kmdvv\") pod \"8b04d115-9b8e-4fb8-ba65-2512d5065070\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " Oct 30 06:43:02.914390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.913388 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-credential-socket\") pod \"8b04d115-9b8e-4fb8-ba65-2512d5065070\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " Oct 30 06:43:02.914390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.913515 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-envoy\") pod \"8b04d115-9b8e-4fb8-ba65-2512d5065070\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " Oct 30 06:43:02.914390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.913547 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-token\" (UniqueName: \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-token\") pod \"8b04d115-9b8e-4fb8-ba65-2512d5065070\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " Oct 30 06:43:02.914390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.913656 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-socket\") pod \"8b04d115-9b8e-4fb8-ba65-2512d5065070\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " Oct 30 06:43:02.914390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.913894 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b04d115-9b8e-4fb8-ba65-2512d5065070-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b04d115-9b8e-4fb8-ba65-2512d5065070-istiod-ca-cert\") pod \"8b04d115-9b8e-4fb8-ba65-2512d5065070\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " Oct 30 06:43:02.914390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.913942 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-podinfo\") pod \"8b04d115-9b8e-4fb8-ba65-2512d5065070\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " Oct 30 06:43:02.914390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.914033 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-data\") pod \"8b04d115-9b8e-4fb8-ba65-2512d5065070\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " Oct 30 06:43:02.914390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.914064 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-certs\") pod \"8b04d115-9b8e-4fb8-ba65-2512d5065070\" (UID: \"8b04d115-9b8e-4fb8-ba65-2512d5065070\") " Oct 30 06:43:02.959359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.915525 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-credential-socket" pod "8b04d115-9b8e-4fb8-ba65-2512d5065070" (UID: "8b04d115-9b8e-4fb8-ba65-2512d5065070"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:43:02.959359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.920951 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b04d115-9b8e-4fb8-ba65-2512d5065070-istiod-ca-cert" pod "8b04d115-9b8e-4fb8-ba65-2512d5065070" (UID: "8b04d115-9b8e-4fb8-ba65-2512d5065070"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:43:02.959359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.921780 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-data" pod "8b04d115-9b8e-4fb8-ba65-2512d5065070" (UID: "8b04d115-9b8e-4fb8-ba65-2512d5065070"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:43:02.959359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.921969 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-certs" pod "8b04d115-9b8e-4fb8-ba65-2512d5065070" (UID: "8b04d115-9b8e-4fb8-ba65-2512d5065070"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:43:02.959359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.922592 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:43:02.959359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.922613 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b04d115-9b8e-4fb8-ba65-2512d5065070-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:43:02.959359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.922627 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:43:02.959359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.922640 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:43:02.959359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.950445 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-socket" pod "8b04d115-9b8e-4fb8-ba65-2512d5065070" (UID: "8b04d115-9b8e-4fb8-ba65-2512d5065070"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:43:02.973599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:02.972524 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-podinfo" pod "8b04d115-9b8e-4fb8-ba65-2512d5065070" (UID: "8b04d115-9b8e-4fb8-ba65-2512d5065070"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:43:03.077322 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.077221 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.077322 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.077265 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.207270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.207152 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-kube-api-access-kmdvv" pod "8b04d115-9b8e-4fb8-ba65-2512d5065070" (UID: "8b04d115-9b8e-4fb8-ba65-2512d5065070"). InnerVolumeSpecName "kube-api-access-kmdvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:43:03.218710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.218639 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-token" pod "8b04d115-9b8e-4fb8-ba65-2512d5065070" (UID: "8b04d115-9b8e-4fb8-ba65-2512d5065070"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:43:03.243531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.243482 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-envoy" pod "8b04d115-9b8e-4fb8-ba65-2512d5065070" (UID: "8b04d115-9b8e-4fb8-ba65-2512d5065070"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:43:03.257351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.251435 5001 generic.go:356] "Generic (PLEG): container finished" podID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerID="9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5" exitCode=0 Oct 30 06:43:03.257351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.251501 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" event={"ID":"8b04d115-9b8e-4fb8-ba65-2512d5065070","Type":"ContainerDied","Data":"9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5"} Oct 30 06:43:03.257351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.251550 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" event={"ID":"8b04d115-9b8e-4fb8-ba65-2512d5065070","Type":"ContainerDied","Data":"f663f06d135db62a4dac1f67d199d2f245ee3420dc0329d33a42082668693516"} Oct 30 06:43:03.257351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.251579 5001 scope.go:122] "RemoveContainer" containerID="9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5" Oct 30 06:43:03.257351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.251751 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm" Oct 30 06:43:03.337793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.337706 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.337793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.337748 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.337793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.337763 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmdvv\" (UniqueName: \"kubernetes.io/projected/8b04d115-9b8e-4fb8-ba65-2512d5065070-kube-api-access-kmdvv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.441370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.441343 5001 scope.go:122] "RemoveContainer" containerID="9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5" Oct 30 06:43:03.442002 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:43:03.441973 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5\": container with ID starting with 9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5 not found: ID does not exist" containerID="9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5" Oct 30 06:43:03.442232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.442203 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5"} err="failed to get container status \"9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5\": rpc error: code = NotFound desc = could not find container \"9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5\": container with ID starting with 9d49e35cb4bc439ef231ba0235047b820796ef48218e87b996cb30024cfac9e5 not found: ID does not exist" Oct 30 06:43:03.445394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.445372 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm"] Oct 30 06:43:03.448730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:03.448709 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-s22zm"] Oct 30 06:43:04.145796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:04.145760 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" path="/var/lib/kubelet/pods/8b04d115-9b8e-4fb8-ba65-2512d5065070/volumes" Oct 30 06:43:06.989781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:06.989747 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:43:07.006115 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:07.006088 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:43:08.272637 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:43:08.272604 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:43:10.981849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:10.981816 5001 scope.go:122] "RemoveContainer" containerID="63ee07ec723d5d81324795d9af1227b9b6d880b64e4799a63fc3936ffd2f792b" Oct 30 06:43:10.996155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:10.996116 5001 scope.go:122] "RemoveContainer" containerID="f73c1e0b95b5930a206798c1112921ede25eb669739ce53c721e0806d9d45410" Oct 30 06:43:25.075522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:25.068404 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7"] Oct 30 06:43:25.095166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:25.095131 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c9c6ee-g4df7"] Oct 30 06:43:26.120561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:26.120527 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c743e3a-4fd3-4320-8efa-b977cadd7338" path="/var/lib/kubelet/pods/8c743e3a-4fd3-4320-8efa-b977cadd7338/volumes" Oct 30 06:43:32.035323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:32.035269 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj"] Oct 30 06:43:32.073499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:32.073462 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047c2e57-js6hj"] Oct 30 06:43:32.124047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:32.124009 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277389bd-3a5e-49ee-8821-6b6f4768086f" path="/var/lib/kubelet/pods/277389bd-3a5e-49ee-8821-6b6f4768086f/volumes" Oct 30 06:43:38.273288 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:43:38.273243 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:43:48.218655 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:48.218610 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:43:56.625716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:56.625679 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92"] Oct 30 06:43:56.626204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:56.625884 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" podUID="646d7efa-8189-43f5-803e-7f9706982768" containerName="istio-proxy" containerID="cri-o://07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641" gracePeriod=30 Oct 30 06:43:56.946831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:56.946734 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c"] Oct 30 06:43:56.947025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:56.946971 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" podUID="81f33b27-3d5e-4a49-91e1-e44a8a5ce874" containerName="istio-proxy" containerID="cri-o://1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c" gracePeriod=30 Oct 30 06:43:56.952289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:56.952261 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58"] Oct 30 06:43:56.952554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:43:56.952524 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" containerID="cri-o://0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2" gracePeriod=30 Oct 30 06:44:01.971957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:01.971924 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:44:02.020613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.020577 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-envoy\") pod \"646d7efa-8189-43f5-803e-7f9706982768\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " Oct 30 06:44:02.020782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.020626 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-certs\") pod \"646d7efa-8189-43f5-803e-7f9706982768\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " Oct 30 06:44:02.020782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.020711 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-kube-api-access-dldkq\" (UniqueName: \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-kube-api-access-dldkq\") pod \"646d7efa-8189-43f5-803e-7f9706982768\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " Oct 30 06:44:02.020910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.020819 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-credential-socket\") pod \"646d7efa-8189-43f5-803e-7f9706982768\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " Oct 30 06:44:02.020910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.020861 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/646d7efa-8189-43f5-803e-7f9706982768-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646d7efa-8189-43f5-803e-7f9706982768-istiod-ca-cert\") pod \"646d7efa-8189-43f5-803e-7f9706982768\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " Oct 30 06:44:02.021034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.020929 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-istio-token\" (UniqueName: \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-istio-token\") pod \"646d7efa-8189-43f5-803e-7f9706982768\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " Oct 30 06:44:02.021034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.020947 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-certs" pod "646d7efa-8189-43f5-803e-7f9706982768" (UID: "646d7efa-8189-43f5-803e-7f9706982768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.021034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.020967 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/646d7efa-8189-43f5-803e-7f9706982768-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646d7efa-8189-43f5-803e-7f9706982768-istio-podinfo\") pod \"646d7efa-8189-43f5-803e-7f9706982768\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " Oct 30 06:44:02.021211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.021078 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-credential-socket" pod "646d7efa-8189-43f5-803e-7f9706982768" (UID: "646d7efa-8189-43f5-803e-7f9706982768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.021211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.021160 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-data\") pod \"646d7efa-8189-43f5-803e-7f9706982768\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " Oct 30 06:44:02.021350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.021226 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-socket\") pod \"646d7efa-8189-43f5-803e-7f9706982768\" (UID: \"646d7efa-8189-43f5-803e-7f9706982768\") " Oct 30 06:44:02.021350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.021274 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646d7efa-8189-43f5-803e-7f9706982768-istiod-ca-cert" pod "646d7efa-8189-43f5-803e-7f9706982768" (UID: "646d7efa-8189-43f5-803e-7f9706982768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:44:02.021726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.021545 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-data" pod "646d7efa-8189-43f5-803e-7f9706982768" (UID: "646d7efa-8189-43f5-803e-7f9706982768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.021864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.021766 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-socket" pod "646d7efa-8189-43f5-803e-7f9706982768" (UID: "646d7efa-8189-43f5-803e-7f9706982768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.022375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.022351 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.022488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.022382 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.022488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.022397 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646d7efa-8189-43f5-803e-7f9706982768-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.022488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.022410 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.022488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.022423 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.023742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.023715 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-kube-api-access-dldkq" pod "646d7efa-8189-43f5-803e-7f9706982768" (UID: "646d7efa-8189-43f5-803e-7f9706982768"). InnerVolumeSpecName "kube-api-access-dldkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:02.024019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.024002 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/646d7efa-8189-43f5-803e-7f9706982768-istio-podinfo" pod "646d7efa-8189-43f5-803e-7f9706982768" (UID: "646d7efa-8189-43f5-803e-7f9706982768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:44:02.024019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.024001 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-istio-token" pod "646d7efa-8189-43f5-803e-7f9706982768" (UID: "646d7efa-8189-43f5-803e-7f9706982768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:02.024194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.024169 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-envoy" pod "646d7efa-8189-43f5-803e-7f9706982768" (UID: "646d7efa-8189-43f5-803e-7f9706982768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.074259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.074224 5001 generic.go:356] "Generic (PLEG): container finished" podID="646d7efa-8189-43f5-803e-7f9706982768" containerID="07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641" exitCode=0 Oct 30 06:44:02.074568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.074421 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" event={"ID":"646d7efa-8189-43f5-803e-7f9706982768","Type":"ContainerDied","Data":"07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641"} Oct 30 06:44:02.074568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.074467 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" event={"ID":"646d7efa-8189-43f5-803e-7f9706982768","Type":"ContainerDied","Data":"62b7fe5ac4d069187cec6ce2844d47ccbac318a93b4cda6e72814c30970874b3"} Oct 30 06:44:02.074568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.074490 5001 scope.go:122] "RemoveContainer" containerID="07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641" Oct 30 06:44:02.074772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.074629 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92" Oct 30 06:44:02.105816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.105784 5001 scope.go:122] "RemoveContainer" containerID="07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641" Oct 30 06:44:02.106268 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:44:02.106201 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641\": container with ID starting with 07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641 not found: ID does not exist" containerID="07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641" Oct 30 06:44:02.106268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.106242 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641"} err="failed to get container status \"07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641\": rpc error: code = NotFound desc = could not find container \"07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641\": container with ID starting with 07a32acbf91756457c5b557e1a7365faad02abea48c0efe5ad14c6cff7a0d641 not found: ID does not exist" Oct 30 06:44:02.127207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.124700 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.127207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.124734 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646d7efa-8189-43f5-803e-7f9706982768-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.127207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.124748 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646d7efa-8189-43f5-803e-7f9706982768-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.127207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.124761 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dldkq\" (UniqueName: \"kubernetes.io/projected/646d7efa-8189-43f5-803e-7f9706982768-kube-api-access-dldkq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.161170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.159688 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92"] Oct 30 06:44:02.169378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.169299 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-kkz92"] Oct 30 06:44:02.310029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.309989 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:44:02.415531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.415493 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:44:02.429483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.429404 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-socket\") pod \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " Oct 30 06:44:02.429483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.429460 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-podinfo\") pod \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " Oct 30 06:44:02.429696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.429542 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-certs\") pod \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " Oct 30 06:44:02.429696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.429608 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istiod-ca-cert\") pod \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " Oct 30 06:44:02.429696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.429680 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-token\" (UniqueName: \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-token\") pod \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " Oct 30 06:44:02.429810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.429719 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-credential-socket\") pod \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " Oct 30 06:44:02.429810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.429744 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-data\") pod \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " Oct 30 06:44:02.429810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.429768 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-kube-api-access-xfmj9\" (UniqueName: \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-kube-api-access-xfmj9\") pod \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " Oct 30 06:44:02.429810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.429793 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-envoy\") pod \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\" (UID: \"7fdd5e60-c7ac-460d-883a-f4e4961f3735\") " Oct 30 06:44:02.430862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.430833 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istiod-ca-cert" pod "7fdd5e60-c7ac-460d-883a-f4e4961f3735" (UID: "7fdd5e60-c7ac-460d-883a-f4e4961f3735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:44:02.431102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.431082 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-socket" pod "7fdd5e60-c7ac-460d-883a-f4e4961f3735" (UID: "7fdd5e60-c7ac-460d-883a-f4e4961f3735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.431737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.431713 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-credential-socket" pod "7fdd5e60-c7ac-460d-883a-f4e4961f3735" (UID: "7fdd5e60-c7ac-460d-883a-f4e4961f3735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.432285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.432205 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-data" pod "7fdd5e60-c7ac-460d-883a-f4e4961f3735" (UID: "7fdd5e60-c7ac-460d-883a-f4e4961f3735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.432285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.432250 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-certs" pod "7fdd5e60-c7ac-460d-883a-f4e4961f3735" (UID: "7fdd5e60-c7ac-460d-883a-f4e4961f3735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.432749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.432729 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-envoy" pod "7fdd5e60-c7ac-460d-883a-f4e4961f3735" (UID: "7fdd5e60-c7ac-460d-883a-f4e4961f3735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.434187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.434158 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-kube-api-access-xfmj9" pod "7fdd5e60-c7ac-460d-883a-f4e4961f3735" (UID: "7fdd5e60-c7ac-460d-883a-f4e4961f3735"). InnerVolumeSpecName "kube-api-access-xfmj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:02.434285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.434237 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-podinfo" pod "7fdd5e60-c7ac-460d-883a-f4e4961f3735" (UID: "7fdd5e60-c7ac-460d-883a-f4e4961f3735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:44:02.434442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.434422 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-token" pod "7fdd5e60-c7ac-460d-883a-f4e4961f3735" (UID: "7fdd5e60-c7ac-460d-883a-f4e4961f3735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:02.531724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.531687 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istiod-ca-cert\") pod \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " Oct 30 06:44:02.531946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.531761 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-data\") pod \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " Oct 30 06:44:02.531946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.531812 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-certs\") pod \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " Oct 30 06:44:02.531946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.531860 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-kube-api-access-hmkbw\" (UniqueName: \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-kube-api-access-hmkbw\") pod \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " Oct 30 06:44:02.531946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.531922 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-socket\") pod \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " Oct 30 06:44:02.532211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.532080 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-credential-socket\") pod \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " Oct 30 06:44:02.532211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.532201 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-envoy\") pod \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " Oct 30 06:44:02.532211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.532203 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istiod-ca-cert" pod "81f33b27-3d5e-4a49-91e1-e44a8a5ce874" (UID: "81f33b27-3d5e-4a49-91e1-e44a8a5ce874"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:44:02.532451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.532295 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-token\" (UniqueName: \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-token\") pod \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " Oct 30 06:44:02.532451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.532376 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-podinfo\") pod \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\" (UID: \"81f33b27-3d5e-4a49-91e1-e44a8a5ce874\") " Oct 30 06:44:02.532451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.532405 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-credential-socket" pod "81f33b27-3d5e-4a49-91e1-e44a8a5ce874" (UID: "81f33b27-3d5e-4a49-91e1-e44a8a5ce874"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.532862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.532735 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-data" pod "81f33b27-3d5e-4a49-91e1-e44a8a5ce874" (UID: "81f33b27-3d5e-4a49-91e1-e44a8a5ce874"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.533005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.532774 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-socket" pod "81f33b27-3d5e-4a49-91e1-e44a8a5ce874" (UID: "81f33b27-3d5e-4a49-91e1-e44a8a5ce874"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.533129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.533107 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-certs" pod "81f33b27-3d5e-4a49-91e1-e44a8a5ce874" (UID: "81f33b27-3d5e-4a49-91e1-e44a8a5ce874"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.533237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.533221 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.533370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.533355 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.533650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.533637 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.533750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.533738 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.533845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.533833 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.533993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.533976 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.534101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.534088 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.534202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.534189 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.534295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.534282 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfmj9\" (UniqueName: \"kubernetes.io/projected/7fdd5e60-c7ac-460d-883a-f4e4961f3735-kube-api-access-xfmj9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.534416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.534401 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.534545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.534533 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdd5e60-c7ac-460d-883a-f4e4961f3735-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.535089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.535061 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-token" pod "81f33b27-3d5e-4a49-91e1-e44a8a5ce874" (UID: "81f33b27-3d5e-4a49-91e1-e44a8a5ce874"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:02.535262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.535230 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-podinfo" pod "81f33b27-3d5e-4a49-91e1-e44a8a5ce874" (UID: "81f33b27-3d5e-4a49-91e1-e44a8a5ce874"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:44:02.535431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.535406 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-kube-api-access-hmkbw" pod "81f33b27-3d5e-4a49-91e1-e44a8a5ce874" (UID: "81f33b27-3d5e-4a49-91e1-e44a8a5ce874"). InnerVolumeSpecName "kube-api-access-hmkbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:02.536266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.536245 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-envoy" pod "81f33b27-3d5e-4a49-91e1-e44a8a5ce874" (UID: "81f33b27-3d5e-4a49-91e1-e44a8a5ce874"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:02.643353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.636688 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.643353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.636732 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.643353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.636749 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.643353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.636762 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.643353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.636775 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.643353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.636790 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmkbw\" (UniqueName: \"kubernetes.io/projected/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-kube-api-access-hmkbw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:02.643353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:02.636806 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f33b27-3d5e-4a49-91e1-e44a8a5ce874-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:03.089891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.089851 5001 generic.go:356] "Generic (PLEG): container finished" podID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerID="0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2" exitCode=0 Oct 30 06:44:03.090558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.089915 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" Oct 30 06:44:03.090558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.089942 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" event={"ID":"7fdd5e60-c7ac-460d-883a-f4e4961f3735","Type":"ContainerDied","Data":"0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2"} Oct 30 06:44:03.090558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.090082 5001 scope.go:122] "RemoveContainer" containerID="0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2" Oct 30 06:44:03.090558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.090211 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58" event={"ID":"7fdd5e60-c7ac-460d-883a-f4e4961f3735","Type":"ContainerDied","Data":"1b42ce9bf182efafb3412b2b222733a0f73741295208c14dee01f56014bcb8ae"} Oct 30 06:44:03.091980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.091954 5001 generic.go:356] "Generic (PLEG): container finished" podID="81f33b27-3d5e-4a49-91e1-e44a8a5ce874" containerID="1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c" exitCode=0 Oct 30 06:44:03.092182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.091976 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" Oct 30 06:44:03.092324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.091958 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" event={"ID":"81f33b27-3d5e-4a49-91e1-e44a8a5ce874","Type":"ContainerDied","Data":"1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c"} Oct 30 06:44:03.092324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.092268 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c" event={"ID":"81f33b27-3d5e-4a49-91e1-e44a8a5ce874","Type":"ContainerDied","Data":"92dc7120d6dace30e463d8da407768723c8544b0927eb397738955e948592338"} Oct 30 06:44:03.107049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.107025 5001 scope.go:122] "RemoveContainer" containerID="29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a" Oct 30 06:44:03.120474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.120447 5001 scope.go:122] "RemoveContainer" containerID="0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2" Oct 30 06:44:03.120804 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:44:03.120776 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2\": container with ID starting with 0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2 not found: ID does not exist" containerID="0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2" Oct 30 06:44:03.120956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.120813 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2"} err="failed to get container status \"0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2\": rpc error: code = NotFound desc = could not find container \"0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2\": container with ID starting with 0d18624a4a3a77544af288631c831e3610cb88c87abdf104f9025829511d27f2 not found: ID does not exist" Oct 30 06:44:03.120956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.120832 5001 scope.go:122] "RemoveContainer" containerID="29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a" Oct 30 06:44:03.121142 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:44:03.121120 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a\": container with ID starting with 29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a not found: ID does not exist" containerID="29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a" Oct 30 06:44:03.121186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.121153 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a"} err="failed to get container status \"29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a\": rpc error: code = NotFound desc = could not find container \"29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a\": container with ID starting with 29e19a87c52508f0633f970646f691495b34270a32719b1797bfa22bfd8ce14a not found: ID does not exist" Oct 30 06:44:03.121186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.121174 5001 scope.go:122] "RemoveContainer" containerID="1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c" Oct 30 06:44:03.133081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.133058 5001 scope.go:122] "RemoveContainer" containerID="1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c" Oct 30 06:44:03.133466 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:44:03.133438 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c\": container with ID starting with 1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c not found: ID does not exist" containerID="1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c" Oct 30 06:44:03.133641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.133476 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c"} err="failed to get container status \"1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c\": rpc error: code = NotFound desc = could not find container \"1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c\": container with ID starting with 1847bf36df58aec1b0f3ccc280afdbba74478c4cd57e0ed33aa625cd9b5fea1c not found: ID does not exist" Oct 30 06:44:03.150102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.150066 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c"] Oct 30 06:44:03.161295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.161263 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-82g8c"] Oct 30 06:44:03.198891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.198848 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58"] Oct 30 06:44:03.203735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:03.203711 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-mgx58"] Oct 30 06:44:04.138599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:04.138565 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="646d7efa-8189-43f5-803e-7f9706982768" path="/var/lib/kubelet/pods/646d7efa-8189-43f5-803e-7f9706982768/volumes" Oct 30 06:44:04.139141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:04.139126 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" path="/var/lib/kubelet/pods/7fdd5e60-c7ac-460d-883a-f4e4961f3735/volumes" Oct 30 06:44:04.139619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:04.139606 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f33b27-3d5e-4a49-91e1-e44a8a5ce874" path="/var/lib/kubelet/pods/81f33b27-3d5e-4a49-91e1-e44a8a5ce874/volumes" Oct 30 06:44:07.844349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.844257 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7"] Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849376 5001 cpu_manager.go:414] "containerMap: removing container" podUID="81f33b27-3d5e-4a49-91e1-e44a8a5ce874" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849403 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f33b27-3d5e-4a49-91e1-e44a8a5ce874" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849440 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849447 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849487 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849494 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849519 5001 cpu_manager.go:414] "containerMap: removing container" podUID="646d7efa-8189-43f5-803e-7f9706982768" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849526 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646d7efa-8189-43f5-803e-7f9706982768" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849546 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849552 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849580 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4d01dd55-de02-4f0c-83d6-303271866adf" containerName="payload-processing" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849587 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d01dd55-de02-4f0c-83d6-303271866adf" containerName="payload-processing" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849627 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3186c702-d417-4137-93ad-81abbb463f5e" containerName="payload-processing" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.849633 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3186c702-d417-4137-93ad-81abbb463f5e" containerName="payload-processing" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.850023 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f33b27-3d5e-4a49-91e1-e44a8a5ce874" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.850051 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.850068 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b04d115-9b8e-4fb8-ba65-2512d5065070" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.850078 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="646d7efa-8189-43f5-803e-7f9706982768" containerName="istio-proxy" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.850100 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d01dd55-de02-4f0c-83d6-303271866adf" containerName="payload-processing" Oct 30 06:44:07.850495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.850125 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3186c702-d417-4137-93ad-81abbb463f5e" containerName="payload-processing" Oct 30 06:44:07.855835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.855812 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:07.865194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.865164 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7"] Oct 30 06:44:07.872098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.872069 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:44:07.872254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.872234 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:44:07.883938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.883915 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:44:07.884064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.883968 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:44:07.927964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.927931 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ghc\" (UniqueName: \"kubernetes.io/projected/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-kube-api-access-d9ghc\") pod \"maas-api-revoke-sub-25598b7b-ww7m7\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:07.928134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.928118 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25598b7b-ww7m7\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:07.928192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:07.928178 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25598b7b-ww7m7\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:08.030177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:08.030141 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ghc\" (UniqueName: \"kubernetes.io/projected/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-kube-api-access-d9ghc\") pod \"maas-api-revoke-sub-25598b7b-ww7m7\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:08.030436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:08.030287 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25598b7b-ww7m7\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:08.030436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:08.030377 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25598b7b-ww7m7\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:08.031073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:08.031051 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25598b7b-ww7m7\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:08.032888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:08.032870 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25598b7b-ww7m7\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:08.055412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:08.055383 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ghc\" (UniqueName: \"kubernetes.io/projected/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-kube-api-access-d9ghc\") pod \"maas-api-revoke-sub-25598b7b-ww7m7\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:08.175657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:08.175626 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:08.276143 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:44:08.276107 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:44:08.488102 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:44:08.488069 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f2b5ed_ff84_40df_9ad7_a1ff3586eecb.slice/crio-6277ddeb92fc084e05d4a1761be55b3da56391363de55d5742eda1cbb30c0d58 WatchSource:0}: Error finding container 6277ddeb92fc084e05d4a1761be55b3da56391363de55d5742eda1cbb30c0d58: Status 404 returned error can't find the container with id 6277ddeb92fc084e05d4a1761be55b3da56391363de55d5742eda1cbb30c0d58 Oct 30 06:44:08.488383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:08.488347 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7"] Oct 30 06:44:09.156665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:09.156568 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" event={"ID":"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb","Type":"ContainerStarted","Data":"8f4d7c4ecaa5ed0dca9f1fb0bd2a8fd6da665eb8598328e3952f037e87aeb918"} Oct 30 06:44:09.156665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:09.156638 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" event={"ID":"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb","Type":"ContainerStarted","Data":"6277ddeb92fc084e05d4a1761be55b3da56391363de55d5742eda1cbb30c0d58"} Oct 30 06:44:09.202118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:09.202064 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" podStartSLOduration=2.202049912 podStartE2EDuration="2.202049912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:44:09.198616529 +0000 UTC m=+3063.194573542" watchObservedRunningTime="2026-10-30 06:44:09.202049912 +0000 UTC m=+3063.198006922" Oct 30 06:44:10.034853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:10.034820 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp"] Oct 30 06:44:10.047402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:10.047370 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2de17a6c-djcwp"] Oct 30 06:44:10.120236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:10.120201 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130ca010-84f5-4c85-a641-3e6456b30222" path="/var/lib/kubelet/pods/130ca010-84f5-4c85-a641-3e6456b30222/volumes" Oct 30 06:44:10.169637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:10.169596 5001 generic.go:356] "Generic (PLEG): container finished" podID="71f2b5ed-ff84-40df-9ad7-a1ff3586eecb" containerID="8f4d7c4ecaa5ed0dca9f1fb0bd2a8fd6da665eb8598328e3952f037e87aeb918" exitCode=0 Oct 30 06:44:10.170152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:10.169667 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" event={"ID":"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb","Type":"ContainerDied","Data":"8f4d7c4ecaa5ed0dca9f1fb0bd2a8fd6da665eb8598328e3952f037e87aeb918"} Oct 30 06:44:11.063635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:11.063596 5001 scope.go:122] "RemoveContainer" containerID="fd57256619446f2d7bd15580465daad0f712b944ffdff6dffc9a5edabb0cc35e" Oct 30 06:44:11.440457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:11.440430 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:11.595252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:11.595220 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-xks-ca\") pod \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " Oct 30 06:44:11.595508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:11.595387 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-kube-api-access-d9ghc\" (UniqueName: \"kubernetes.io/projected/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-kube-api-access-d9ghc\") pod \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " Oct 30 06:44:11.595508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:11.595495 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-openshift-ca\") pod \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\" (UID: \"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb\") " Oct 30 06:44:11.595874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:11.595847 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-openshift-ca" pod "71f2b5ed-ff84-40df-9ad7-a1ff3586eecb" (UID: "71f2b5ed-ff84-40df-9ad7-a1ff3586eecb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:44:11.596371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:11.596348 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:11.597826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:11.597801 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-kube-api-access-d9ghc" pod "71f2b5ed-ff84-40df-9ad7-a1ff3586eecb" (UID: "71f2b5ed-ff84-40df-9ad7-a1ff3586eecb"). InnerVolumeSpecName "kube-api-access-d9ghc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:11.598027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:11.598010 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-xks-ca" pod "71f2b5ed-ff84-40df-9ad7-a1ff3586eecb" (UID: "71f2b5ed-ff84-40df-9ad7-a1ff3586eecb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:44:11.698084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:11.697995 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:11.698084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:11.698024 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9ghc\" (UniqueName: \"kubernetes.io/projected/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb-kube-api-access-d9ghc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:12.200188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:12.200142 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" event={"ID":"71f2b5ed-ff84-40df-9ad7-a1ff3586eecb","Type":"ContainerDied","Data":"6277ddeb92fc084e05d4a1761be55b3da56391363de55d5742eda1cbb30c0d58"} Oct 30 06:44:12.200434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:12.200215 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6277ddeb92fc084e05d4a1761be55b3da56391363de55d5742eda1cbb30c0d58" Oct 30 06:44:12.200434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:12.200292 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7" Oct 30 06:44:19.784220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.776398 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj"] Oct 30 06:44:19.788325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.786189 5001 cpu_manager.go:414] "containerMap: removing container" podUID="71f2b5ed-ff84-40df-9ad7-a1ff3586eecb" containerName="revoke-keys" Oct 30 06:44:19.788325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.786221 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f2b5ed-ff84-40df-9ad7-a1ff3586eecb" containerName="revoke-keys" Oct 30 06:44:19.788325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.786447 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f2b5ed-ff84-40df-9ad7-a1ff3586eecb" containerName="revoke-keys" Oct 30 06:44:19.788325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.786473 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdd5e60-c7ac-460d-883a-f4e4961f3735" containerName="istio-proxy" Oct 30 06:44:19.800712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.800682 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:19.806668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.806600 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj"] Oct 30 06:44:19.871275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.871235 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-provider-credentials\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:19.871517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.871289 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64sn9\" (UniqueName: \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-kube-api-access-64sn9\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:19.871579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.871522 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-routing-overlay-volume\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:19.871773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.871748 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-plugins-config-volume\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:19.973019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.972976 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-provider-credentials\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:19.973019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.973033 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64sn9\" (UniqueName: \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-kube-api-access-64sn9\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:19.973328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.973190 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-routing-overlay-volume\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:19.973406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.973385 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-plugins-config-volume\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:19.974161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.974130 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-routing-overlay-volume\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:19.974349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.974255 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-plugins-config-volume\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:19.976035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:19.976005 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-provider-credentials\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:20.000110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:20.000077 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64sn9\" (UniqueName: \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-kube-api-access-64sn9\") pod \"payload-processing-external-model-65dff8f9bc-kxmrj\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:20.140329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:20.139685 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:20.739633 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:44:20.739597 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121c8afc_303a_449b_81dc_8c3e795b598d.slice/crio-a578de998caab64e0a43fc134994bcccf02c867b9fa32abb945dd5d3d49f866c WatchSource:0}: Error finding container a578de998caab64e0a43fc134994bcccf02c867b9fa32abb945dd5d3d49f866c: Status 404 returned error can't find the container with id a578de998caab64e0a43fc134994bcccf02c867b9fa32abb945dd5d3d49f866c Oct 30 06:44:20.739780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:20.739658 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj"] Oct 30 06:44:21.388192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:21.388147 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" event={"ID":"121c8afc-303a-449b-81dc-8c3e795b598d","Type":"ContainerStarted","Data":"198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f"} Oct 30 06:44:21.388695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:21.388216 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:21.388695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:21.388241 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" event={"ID":"121c8afc-303a-449b-81dc-8c3e795b598d","Type":"ContainerStarted","Data":"a578de998caab64e0a43fc134994bcccf02c867b9fa32abb945dd5d3d49f866c"} Oct 30 06:44:21.434548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:21.434494 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" podStartSLOduration=2.434480056 podStartE2EDuration="2.434480056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:44:21.430915196 +0000 UTC m=+3075.426872207" watchObservedRunningTime="2026-10-30 06:44:21.434480056 +0000 UTC m=+3075.430437065" Oct 30 06:44:22.036273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.036234 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt"] Oct 30 06:44:22.049694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.049643 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695c9bf9-7v9zt"] Oct 30 06:44:22.122658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.122628 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb485d73-259f-44ca-be8f-c015ba4e9c80" path="/var/lib/kubelet/pods/eb485d73-259f-44ca-be8f-c015ba4e9c80/volumes" Oct 30 06:44:22.335065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.334981 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p"] Oct 30 06:44:22.341221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.341199 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:44:22.356934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.356899 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-f08338\"/\"kube-root-ca.crt\"" Oct 30 06:44:22.356934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.356915 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p"] Oct 30 06:44:22.357119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.356953 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-f08338\"/\"openshift-service-ca.crt\"" Oct 30 06:44:22.373165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.373136 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-f08338\"/\"e2e-usage-sim-b3afd6-cert\"" Oct 30 06:44:22.373359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.373199 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-f08338\"/\"default-dockercfg-p94xr\"" Oct 30 06:44:22.440674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.440639 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7c3862d1-263a-42e6-a01f-90b4ee0629dc-tls\") pod \"e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p\" (UID: \"7c3862d1-263a-42e6-a01f-90b4ee0629dc\") " pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:44:22.441109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.440762 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57dsw\" (UniqueName: \"kubernetes.io/projected/7c3862d1-263a-42e6-a01f-90b4ee0629dc-kube-api-access-57dsw\") pod \"e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p\" (UID: \"7c3862d1-263a-42e6-a01f-90b4ee0629dc\") " pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:44:22.546412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.546375 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57dsw\" (UniqueName: \"kubernetes.io/projected/7c3862d1-263a-42e6-a01f-90b4ee0629dc-kube-api-access-57dsw\") pod \"e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p\" (UID: \"7c3862d1-263a-42e6-a01f-90b4ee0629dc\") " pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:44:22.546963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.546939 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7c3862d1-263a-42e6-a01f-90b4ee0629dc-tls\") pod \"e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p\" (UID: \"7c3862d1-263a-42e6-a01f-90b4ee0629dc\") " pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:44:22.588381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.586658 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57dsw\" (UniqueName: \"kubernetes.io/projected/7c3862d1-263a-42e6-a01f-90b4ee0629dc-kube-api-access-57dsw\") pod \"e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p\" (UID: \"7c3862d1-263a-42e6-a01f-90b4ee0629dc\") " pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:44:22.589250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.589214 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7c3862d1-263a-42e6-a01f-90b4ee0629dc-tls\") pod \"e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p\" (UID: \"7c3862d1-263a-42e6-a01f-90b4ee0629dc\") " pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:44:22.659120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.659087 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:44:22.983672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:22.983641 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p"] Oct 30 06:44:22.985300 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:44:22.985267 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c3862d1_263a_42e6_a01f_90b4ee0629dc.slice/crio-324caed7686c85e03b466d0d4559e67adf57edad83d9c4af9cb2db66421abc48 WatchSource:0}: Error finding container 324caed7686c85e03b466d0d4559e67adf57edad83d9c4af9cb2db66421abc48: Status 404 returned error can't find the container with id 324caed7686c85e03b466d0d4559e67adf57edad83d9c4af9cb2db66421abc48 Oct 30 06:44:23.410622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:23.410588 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" event={"ID":"7c3862d1-263a-42e6-a01f-90b4ee0629dc","Type":"ContainerStarted","Data":"56403ee78f639d3062ea035921811d9436ee3d7ed6fd141a37894a230091f714"} Oct 30 06:44:23.410622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:23.410630 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" event={"ID":"7c3862d1-263a-42e6-a01f-90b4ee0629dc","Type":"ContainerStarted","Data":"324caed7686c85e03b466d0d4559e67adf57edad83d9c4af9cb2db66421abc48"} Oct 30 06:44:23.410902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:23.410748 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:44:23.416142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:23.416119 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:44:23.498784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:23.498729 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" podStartSLOduration=1.498714687 podStartE2EDuration="1.498714687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:44:23.495395363 +0000 UTC m=+3077.491352373" watchObservedRunningTime="2026-10-30 06:44:23.498714687 +0000 UTC m=+3077.494671697" Oct 30 06:44:30.952990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:30.952959 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4"] Oct 30 06:44:30.963152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:30.963115 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:30.979763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:30.979733 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4"] Oct 30 06:44:30.985874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:30.985778 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"agc-cred-a32003702ba8\"" Oct 30 06:44:30.986280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:30.986253 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ad1661\"" Oct 30 06:44:30.986473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:30.986432 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"payload-processing-external-model-e2e--068888f6-dockercfg-6lqll\"" Oct 30 06:44:30.986574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:30.986503 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"routing-overlay\"" Oct 30 06:44:30.986641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:30.986587 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"openshift-service-ca.crt\"" Oct 30 06:44:30.991551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:30.991523 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"kube-root-ca.crt\"" Oct 30 06:44:31.070544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.070506 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.070749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.070638 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ncfr\" (UniqueName: \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-kube-api-access-7ncfr\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.070864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.070847 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.070934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.070899 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.179660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.179612 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.180944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.180922 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.181945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.181923 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.182179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.182162 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ncfr\" (UniqueName: \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-kube-api-access-7ncfr\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.182796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.180818 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.183378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.181780 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.190633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.186221 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.226390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.215100 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ncfr\" (UniqueName: \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-kube-api-access-7ncfr\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.286328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.285896 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:31.683279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.683237 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4"] Oct 30 06:44:31.688580 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:44:31.688552 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd4206d_f00c_48ad_97b3_e0feedf8e7ba.slice/crio-a8176ace67ada5ae43f0cab0cf15e5aa172e5e360c019e7e6b00edc971de9577 WatchSource:0}: Error finding container a8176ace67ada5ae43f0cab0cf15e5aa172e5e360c019e7e6b00edc971de9577: Status 404 returned error can't find the container with id a8176ace67ada5ae43f0cab0cf15e5aa172e5e360c019e7e6b00edc971de9577 Oct 30 06:44:32.000410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:31.994228 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks"] Oct 30 06:44:32.005371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.003533 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.016985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.016953 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks"] Oct 30 06:44:32.019218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.019195 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"agc-cred-27584b71e6c2\"" Oct 30 06:44:32.019435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.019222 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"routing-overlay\"" Oct 30 06:44:32.023584 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.023558 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"openshift-service-ca.crt\"" Oct 30 06:44:32.023717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.023626 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"kube-root-ca.crt\"" Oct 30 06:44:32.023717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.023573 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6d1778\"" Oct 30 06:44:32.023717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.023654 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"payload-processing-external-model-e2e--c777bb0f-dockercfg-v9bbq\"" Oct 30 06:44:32.102529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.102489 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.102742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.102537 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.102742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.102664 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.102961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.102942 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx88d\" (UniqueName: \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-kube-api-access-jx88d\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.213330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.209882 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.213330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.210283 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx88d\" (UniqueName: \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-kube-api-access-jx88d\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.213330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.210435 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.213330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.210484 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.213330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.212463 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.226331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.221780 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.242330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.233557 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.268707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.267980 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx88d\" (UniqueName: \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-kube-api-access-jx88d\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.324573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.324532 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:32.410364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.407333 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:44:32.498484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.497799 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58946db44c-tdkhz"] Oct 30 06:44:32.507976 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.507921 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" event={"ID":"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba","Type":"ContainerStarted","Data":"00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d"} Oct 30 06:44:32.508167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.507992 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" event={"ID":"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba","Type":"ContainerStarted","Data":"a8176ace67ada5ae43f0cab0cf15e5aa172e5e360c019e7e6b00edc971de9577"} Oct 30 06:44:32.508167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.508083 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:32.612678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.612602 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" podStartSLOduration=2.612581225 podStartE2EDuration="2.612581225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:44:32.611865506 +0000 UTC m=+3086.607822534" watchObservedRunningTime="2026-10-30 06:44:32.612581225 +0000 UTC m=+3086.608538236" Oct 30 06:44:32.721355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:32.717484 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks"] Oct 30 06:44:32.738192 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:44:32.738107 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d41c70e_ac2f_47c9_9c71_3adf2505b04b.slice/crio-360e1f2615e36511f914feb1d37df300c2998f872e2682aae10a4ab67df96633 WatchSource:0}: Error finding container 360e1f2615e36511f914feb1d37df300c2998f872e2682aae10a4ab67df96633: Status 404 returned error can't find the container with id 360e1f2615e36511f914feb1d37df300c2998f872e2682aae10a4ab67df96633 Oct 30 06:44:33.518942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:33.518898 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" event={"ID":"6d41c70e-ac2f-47c9-9c71-3adf2505b04b","Type":"ContainerStarted","Data":"fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0"} Oct 30 06:44:33.519415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:33.518964 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:33.519415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:33.518988 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" event={"ID":"6d41c70e-ac2f-47c9-9c71-3adf2505b04b","Type":"ContainerStarted","Data":"360e1f2615e36511f914feb1d37df300c2998f872e2682aae10a4ab67df96633"} Oct 30 06:44:33.563008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:33.562959 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" podStartSLOduration=2.562943514 podStartE2EDuration="2.562943514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:44:33.562494231 +0000 UTC m=+3087.558451242" watchObservedRunningTime="2026-10-30 06:44:33.562943514 +0000 UTC m=+3087.558900527" Oct 30 06:44:34.035615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:34.035568 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j"] Oct 30 06:44:34.040621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:34.040594 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f578085a-zcx9j"] Oct 30 06:44:34.122958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:34.122901 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7ad224-962d-45ab-b0ca-96c5240a14bf" path="/var/lib/kubelet/pods/7a7ad224-962d-45ab-b0ca-96c5240a14bf/volumes" Oct 30 06:44:37.498420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:37.498355 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" podUID="15fbee68-0311-4e52-bc0b-a64bf7b161ca" containerName="payload-processing" containerID="cri-o://2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1" gracePeriod=25 Oct 30 06:44:37.680987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:37.680951 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:44:37.852841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:37.852816 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:44:38.031703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.031671 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-plugins-config-volume\") pod \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " Oct 30 06:44:38.032001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.031867 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-kube-api-access-p45hp\" (UniqueName: \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-kube-api-access-p45hp\") pod \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " Oct 30 06:44:38.032001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.031911 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-provider-credentials\") pod \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " Oct 30 06:44:38.032165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.032048 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-routing-overlay-volume\") pod \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\" (UID: \"15fbee68-0311-4e52-bc0b-a64bf7b161ca\") " Oct 30 06:44:38.032165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.032135 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-plugins-config-volume" pod "15fbee68-0311-4e52-bc0b-a64bf7b161ca" (UID: "15fbee68-0311-4e52-bc0b-a64bf7b161ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:44:38.032444 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.032418 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-routing-overlay-volume" pod "15fbee68-0311-4e52-bc0b-a64bf7b161ca" (UID: "15fbee68-0311-4e52-bc0b-a64bf7b161ca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:44:38.032773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.032749 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:38.032773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.032776 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fbee68-0311-4e52-bc0b-a64bf7b161ca-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:38.034451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.034427 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-provider-credentials" pod "15fbee68-0311-4e52-bc0b-a64bf7b161ca" (UID: "15fbee68-0311-4e52-bc0b-a64bf7b161ca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:38.034529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.034499 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-kube-api-access-p45hp" pod "15fbee68-0311-4e52-bc0b-a64bf7b161ca" (UID: "15fbee68-0311-4e52-bc0b-a64bf7b161ca"). InnerVolumeSpecName "kube-api-access-p45hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:38.133855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.133830 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p45hp\" (UniqueName: \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-kube-api-access-p45hp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:38.133855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.133857 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15fbee68-0311-4e52-bc0b-a64bf7b161ca-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:38.272818 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:44:38.272782 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:44:38.566759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.566675 5001 generic.go:356] "Generic (PLEG): container finished" podID="15fbee68-0311-4e52-bc0b-a64bf7b161ca" containerID="2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1" exitCode=0 Oct 30 06:44:38.566759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.566742 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" Oct 30 06:44:38.567235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.566757 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" event={"ID":"15fbee68-0311-4e52-bc0b-a64bf7b161ca","Type":"ContainerDied","Data":"2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1"} Oct 30 06:44:38.567235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.566859 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58946db44c-tdkhz" event={"ID":"15fbee68-0311-4e52-bc0b-a64bf7b161ca","Type":"ContainerDied","Data":"52823f63ed668f4051d92753615e5020d6108fd37a1bc69aa6ea64667789a08d"} Oct 30 06:44:38.567235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.566892 5001 scope.go:122] "RemoveContainer" containerID="2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1" Oct 30 06:44:38.581079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.581059 5001 scope.go:122] "RemoveContainer" containerID="2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1" Oct 30 06:44:38.581431 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:44:38.581405 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1\": container with ID starting with 2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1 not found: ID does not exist" containerID="2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1" Oct 30 06:44:38.581541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.581442 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1"} err="failed to get container status \"2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1\": rpc error: code = NotFound desc = could not find container \"2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1\": container with ID starting with 2e97406396adbc3254c2f2444681b697a8d664abeec1063beabd092ff30e55b1 not found: ID does not exist" Oct 30 06:44:38.625990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.625949 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58946db44c-tdkhz"] Oct 30 06:44:38.638625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:38.638594 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-58946db44c-tdkhz"] Oct 30 06:44:40.128330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:40.128013 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15fbee68-0311-4e52-bc0b-a64bf7b161ca" path="/var/lib/kubelet/pods/15fbee68-0311-4e52-bc0b-a64bf7b161ca/volumes" Oct 30 06:44:42.695313 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.695261 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb"] Oct 30 06:44:42.696233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.696212 5001 cpu_manager.go:414] "containerMap: removing container" podUID="15fbee68-0311-4e52-bc0b-a64bf7b161ca" containerName="payload-processing" Oct 30 06:44:42.696290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.696234 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15fbee68-0311-4e52-bc0b-a64bf7b161ca" containerName="payload-processing" Oct 30 06:44:42.696384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.696374 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="15fbee68-0311-4e52-bc0b-a64bf7b161ca" containerName="payload-processing" Oct 30 06:44:42.701809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.701787 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:42.720594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.720568 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb"] Oct 30 06:44:42.726347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.726263 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:44:42.726347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.726284 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:44:42.726347 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.726345 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:44:42.730403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.730374 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:44:42.742477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.742423 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25fb5c9-2w6rb\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:42.742676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.742648 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25fb5c9-2w6rb\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:42.742940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.742913 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f4sf\" (UniqueName: \"kubernetes.io/projected/595bb0e9-db79-4e9e-9a95-752eb8435718-kube-api-access-7f4sf\") pod \"maas-api-revoke-sub-d25fb5c9-2w6rb\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:42.844865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.844830 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25fb5c9-2w6rb\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:42.845072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.845050 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f4sf\" (UniqueName: \"kubernetes.io/projected/595bb0e9-db79-4e9e-9a95-752eb8435718-kube-api-access-7f4sf\") pod \"maas-api-revoke-sub-d25fb5c9-2w6rb\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:42.845180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.845165 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25fb5c9-2w6rb\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:42.845886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.845861 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25fb5c9-2w6rb\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:42.847584 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.847558 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25fb5c9-2w6rb\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:42.870353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:42.870288 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f4sf\" (UniqueName: \"kubernetes.io/projected/595bb0e9-db79-4e9e-9a95-752eb8435718-kube-api-access-7f4sf\") pod \"maas-api-revoke-sub-d25fb5c9-2w6rb\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:43.021011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:43.020920 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:43.410672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:43.393493 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb"] Oct 30 06:44:43.522735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:43.522707 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:44:43.778729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:43.778700 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" event={"ID":"595bb0e9-db79-4e9e-9a95-752eb8435718","Type":"ContainerStarted","Data":"b91a303108aeadd0e292acff8ae68d75a1bec8cd964c77c253efcedadbb02a64"} Oct 30 06:44:44.531385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:44.531350 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:44:44.790263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:44.790166 5001 generic.go:356] "Generic (PLEG): container finished" podID="595bb0e9-db79-4e9e-9a95-752eb8435718" containerID="1d2419662c93b0a81b8c25928dc14d08302a16f95eb83d72c29e344b37312936" exitCode=0 Oct 30 06:44:44.790263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:44.790252 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" event={"ID":"595bb0e9-db79-4e9e-9a95-752eb8435718","Type":"ContainerDied","Data":"1d2419662c93b0a81b8c25928dc14d08302a16f95eb83d72c29e344b37312936"} Oct 30 06:44:45.990128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:45.990099 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:46.112633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.112593 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-openshift-ca\") pod \"595bb0e9-db79-4e9e-9a95-752eb8435718\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " Oct 30 06:44:46.112846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.112642 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595bb0e9-db79-4e9e-9a95-752eb8435718-kube-api-access-7f4sf\" (UniqueName: \"kubernetes.io/projected/595bb0e9-db79-4e9e-9a95-752eb8435718-kube-api-access-7f4sf\") pod \"595bb0e9-db79-4e9e-9a95-752eb8435718\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " Oct 30 06:44:46.112846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.112703 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-xks-ca\") pod \"595bb0e9-db79-4e9e-9a95-752eb8435718\" (UID: \"595bb0e9-db79-4e9e-9a95-752eb8435718\") " Oct 30 06:44:46.113048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.113020 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-openshift-ca" pod "595bb0e9-db79-4e9e-9a95-752eb8435718" (UID: "595bb0e9-db79-4e9e-9a95-752eb8435718"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:44:46.113645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.113623 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:46.115105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.115082 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-xks-ca" pod "595bb0e9-db79-4e9e-9a95-752eb8435718" (UID: "595bb0e9-db79-4e9e-9a95-752eb8435718"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:44:46.115480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.115453 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595bb0e9-db79-4e9e-9a95-752eb8435718-kube-api-access-7f4sf" pod "595bb0e9-db79-4e9e-9a95-752eb8435718" (UID: "595bb0e9-db79-4e9e-9a95-752eb8435718"). InnerVolumeSpecName "kube-api-access-7f4sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:46.215451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.215419 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595bb0e9-db79-4e9e-9a95-752eb8435718-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:46.215451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.215446 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f4sf\" (UniqueName: \"kubernetes.io/projected/595bb0e9-db79-4e9e-9a95-752eb8435718-kube-api-access-7f4sf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:46.810231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.810205 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" Oct 30 06:44:46.810452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.810238 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb" event={"ID":"595bb0e9-db79-4e9e-9a95-752eb8435718","Type":"ContainerDied","Data":"b91a303108aeadd0e292acff8ae68d75a1bec8cd964c77c253efcedadbb02a64"} Oct 30 06:44:46.810452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:46.810283 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b91a303108aeadd0e292acff8ae68d75a1bec8cd964c77c253efcedadbb02a64" Oct 30 06:44:53.488262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.488223 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc"] Oct 30 06:44:53.489075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.489060 5001 cpu_manager.go:414] "containerMap: removing container" podUID="595bb0e9-db79-4e9e-9a95-752eb8435718" containerName="revoke-keys" Oct 30 06:44:53.489128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.489077 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595bb0e9-db79-4e9e-9a95-752eb8435718" containerName="revoke-keys" Oct 30 06:44:53.489222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.489212 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="595bb0e9-db79-4e9e-9a95-752eb8435718" containerName="revoke-keys" Oct 30 06:44:53.493504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.493482 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:53.511356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.511324 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc"] Oct 30 06:44:53.516717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.516674 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:44:53.516908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.516686 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:44:53.516908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.516725 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:44:53.517112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.516804 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:44:53.640478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.640435 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhkzc\" (UniqueName: \"kubernetes.io/projected/4345edf9-316c-4034-9bd2-9daba4b25b10-kube-api-access-rhkzc\") pod \"maas-api-revoke-sub-eb4ae34f-f7wmc\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:53.640674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.640503 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4ae34f-f7wmc\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:53.640729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.640695 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4ae34f-f7wmc\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:53.742727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.742630 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhkzc\" (UniqueName: \"kubernetes.io/projected/4345edf9-316c-4034-9bd2-9daba4b25b10-kube-api-access-rhkzc\") pod \"maas-api-revoke-sub-eb4ae34f-f7wmc\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:53.742727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.742716 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4ae34f-f7wmc\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:53.742989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.742959 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4ae34f-f7wmc\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:53.743944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.743913 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4ae34f-f7wmc\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:53.747513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.747488 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4ae34f-f7wmc\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:53.770229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.770188 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhkzc\" (UniqueName: \"kubernetes.io/projected/4345edf9-316c-4034-9bd2-9daba4b25b10-kube-api-access-rhkzc\") pod \"maas-api-revoke-sub-eb4ae34f-f7wmc\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:53.809739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:53.809701 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:54.178183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:54.178148 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc"] Oct 30 06:44:54.911627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:54.911589 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" event={"ID":"4345edf9-316c-4034-9bd2-9daba4b25b10","Type":"ContainerStarted","Data":"1f78947dae3abd4f94bea205139782befac6c8a5a37a7d392364d597ae995e4a"} Oct 30 06:44:54.912114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:54.911640 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" event={"ID":"4345edf9-316c-4034-9bd2-9daba4b25b10","Type":"ContainerStarted","Data":"4ccf8d0c372b1f4e8aea3b80b654d5b8d52c54216a7225b2bf623e62505a7628"} Oct 30 06:44:54.971105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:54.971031 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" podStartSLOduration=1.971009733 podStartE2EDuration="1.971009733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:44:54.95439638 +0000 UTC m=+3108.950353390" watchObservedRunningTime="2026-10-30 06:44:54.971009733 +0000 UTC m=+3108.966966744" Oct 30 06:44:55.922296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:55.922260 5001 generic.go:356] "Generic (PLEG): container finished" podID="4345edf9-316c-4034-9bd2-9daba4b25b10" containerID="1f78947dae3abd4f94bea205139782befac6c8a5a37a7d392364d597ae995e4a" exitCode=0 Oct 30 06:44:55.922788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:55.922336 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" event={"ID":"4345edf9-316c-4034-9bd2-9daba4b25b10","Type":"ContainerDied","Data":"1f78947dae3abd4f94bea205139782befac6c8a5a37a7d392364d597ae995e4a"} Oct 30 06:44:57.129971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.129942 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:57.239504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.239406 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-xks-ca\") pod \"4345edf9-316c-4034-9bd2-9daba4b25b10\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " Oct 30 06:44:57.239504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.239461 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-openshift-ca\") pod \"4345edf9-316c-4034-9bd2-9daba4b25b10\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " Oct 30 06:44:57.239762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.239529 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4345edf9-316c-4034-9bd2-9daba4b25b10-kube-api-access-rhkzc\" (UniqueName: \"kubernetes.io/projected/4345edf9-316c-4034-9bd2-9daba4b25b10-kube-api-access-rhkzc\") pod \"4345edf9-316c-4034-9bd2-9daba4b25b10\" (UID: \"4345edf9-316c-4034-9bd2-9daba4b25b10\") " Oct 30 06:44:57.239867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.239839 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-openshift-ca" pod "4345edf9-316c-4034-9bd2-9daba4b25b10" (UID: "4345edf9-316c-4034-9bd2-9daba4b25b10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:44:57.241014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.240990 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:57.242033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.242005 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-xks-ca" pod "4345edf9-316c-4034-9bd2-9daba4b25b10" (UID: "4345edf9-316c-4034-9bd2-9daba4b25b10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:44:57.242202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.242175 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4345edf9-316c-4034-9bd2-9daba4b25b10-kube-api-access-rhkzc" pod "4345edf9-316c-4034-9bd2-9daba4b25b10" (UID: "4345edf9-316c-4034-9bd2-9daba4b25b10"). InnerVolumeSpecName "kube-api-access-rhkzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:57.342725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.342694 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4345edf9-316c-4034-9bd2-9daba4b25b10-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:57.342725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.342720 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhkzc\" (UniqueName: \"kubernetes.io/projected/4345edf9-316c-4034-9bd2-9daba4b25b10-kube-api-access-rhkzc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:44:57.944505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.944475 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" Oct 30 06:44:57.944505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.944498 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc" event={"ID":"4345edf9-316c-4034-9bd2-9daba4b25b10","Type":"ContainerDied","Data":"4ccf8d0c372b1f4e8aea3b80b654d5b8d52c54216a7225b2bf623e62505a7628"} Oct 30 06:44:57.944768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:44:57.944548 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ccf8d0c372b1f4e8aea3b80b654d5b8d52c54216a7225b2bf623e62505a7628" Oct 30 06:45:00.146964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.146906 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw"] Oct 30 06:45:00.147924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.147893 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4345edf9-316c-4034-9bd2-9daba4b25b10" containerName="revoke-keys" Oct 30 06:45:00.147974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.147925 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4345edf9-316c-4034-9bd2-9daba4b25b10" containerName="revoke-keys" Oct 30 06:45:00.148049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.148039 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4345edf9-316c-4034-9bd2-9daba4b25b10" containerName="revoke-keys" Oct 30 06:45:00.152200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.152174 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" Oct 30 06:45:00.152392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.152248 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq"] Oct 30 06:45:00.168725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.168689 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" Oct 30 06:45:00.171772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.171743 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:45:00.176901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.176872 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs"] Oct 30 06:45:00.177361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.177334 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:45:00.177749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.177728 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:45:00.201570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.200881 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" Oct 30 06:45:00.212784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.210586 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf"] Oct 30 06:45:00.220565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.218364 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" Oct 30 06:45:00.236972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.236618 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzv7\" (UniqueName: \"kubernetes.io/projected/70dfdbc8-cb44-4270-8108-11dddce181b9-kube-api-access-kzzv7\") pod \"maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw\" (UID: \"70dfdbc8-cb44-4270-8108-11dddce181b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" Oct 30 06:45:00.236972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.236830 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70dfdbc8-cb44-4270-8108-11dddce181b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw\" (UID: \"70dfdbc8-cb44-4270-8108-11dddce181b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" Oct 30 06:45:00.248843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.248809 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7"] Oct 30 06:45:00.265555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.262464 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8"] Oct 30 06:45:00.265555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.263230 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" Oct 30 06:45:00.294533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.294494 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs"] Oct 30 06:45:00.294707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.294620 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" Oct 30 06:45:00.306329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.306287 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk"] Oct 30 06:45:00.312178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.312147 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" Oct 30 06:45:00.315425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.314034 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw"] Oct 30 06:45:00.323276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.322969 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn"] Oct 30 06:45:00.332372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.330111 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" Oct 30 06:45:00.332372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.332122 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq"] Oct 30 06:45:00.338438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.338143 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7"] Oct 30 06:45:00.338603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.338509 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjftg\" (UniqueName: \"kubernetes.io/projected/62f0f671-aaa1-4a3f-a267-19d668b62f80-kube-api-access-xjftg\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs\" (UID: \"62f0f671-aaa1-4a3f-a267-19d668b62f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" Oct 30 06:45:00.338668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.338632 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70dfdbc8-cb44-4270-8108-11dddce181b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw\" (UID: \"70dfdbc8-cb44-4270-8108-11dddce181b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" Oct 30 06:45:00.339682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.338771 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f0f671-aaa1-4a3f-a267-19d668b62f80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs\" (UID: \"62f0f671-aaa1-4a3f-a267-19d668b62f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" Oct 30 06:45:00.339682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.338849 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvm6v\" (UniqueName: \"kubernetes.io/projected/2ced181d-88b9-407f-8663-01b2f28072c2-kube-api-access-vvm6v\") pod \"maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf\" (UID: \"2ced181d-88b9-407f-8663-01b2f28072c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" Oct 30 06:45:00.339682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.338878 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ced181d-88b9-407f-8663-01b2f28072c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf\" (UID: \"2ced181d-88b9-407f-8663-01b2f28072c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" Oct 30 06:45:00.339682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.338909 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq\" (UID: \"50b32463-c8ed-417f-8718-c5ab4fd2f1b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" Oct 30 06:45:00.339682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.338938 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2flf6\" (UniqueName: \"kubernetes.io/projected/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-kube-api-access-2flf6\") pod \"maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq\" (UID: \"50b32463-c8ed-417f-8718-c5ab4fd2f1b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" Oct 30 06:45:00.339682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.339072 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzv7\" (UniqueName: \"kubernetes.io/projected/70dfdbc8-cb44-4270-8108-11dddce181b9-kube-api-access-kzzv7\") pod \"maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw\" (UID: \"70dfdbc8-cb44-4270-8108-11dddce181b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" Oct 30 06:45:00.339682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.339417 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70dfdbc8-cb44-4270-8108-11dddce181b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw\" (UID: \"70dfdbc8-cb44-4270-8108-11dddce181b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" Oct 30 06:45:00.346379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.345174 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf"] Oct 30 06:45:00.350538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.350515 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq"] Oct 30 06:45:00.355590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.355564 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" Oct 30 06:45:00.356012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.355711 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt"] Oct 30 06:45:00.363805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.363782 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" Oct 30 06:45:00.367776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.367676 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzv7\" (UniqueName: \"kubernetes.io/projected/70dfdbc8-cb44-4270-8108-11dddce181b9-kube-api-access-kzzv7\") pod \"maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw\" (UID: \"70dfdbc8-cb44-4270-8108-11dddce181b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" Oct 30 06:45:00.368440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.368420 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk"] Oct 30 06:45:00.376176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.376154 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn"] Oct 30 06:45:00.384811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.384777 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw"] Oct 30 06:45:00.390324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.390281 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" Oct 30 06:45:00.391708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.391681 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x"] Oct 30 06:45:00.397359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.397261 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" Oct 30 06:45:00.397753 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.397731 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8"] Oct 30 06:45:00.403533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.403508 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl"] Oct 30 06:45:00.409835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.409812 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" Oct 30 06:45:00.409938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.409811 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq"] Oct 30 06:45:00.416387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.416288 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt"] Oct 30 06:45:00.425155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.425103 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw"] Oct 30 06:45:00.431706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.431677 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5"] Oct 30 06:45:00.438628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.438605 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h"] Oct 30 06:45:00.438749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.438730 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" Oct 30 06:45:00.440093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440070 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjftg\" (UniqueName: \"kubernetes.io/projected/62f0f671-aaa1-4a3f-a267-19d668b62f80-kube-api-access-xjftg\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs\" (UID: \"62f0f671-aaa1-4a3f-a267-19d668b62f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" Oct 30 06:45:00.440206 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440169 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7\" (UID: \"12f95efa-c8f1-4e3a-a5db-cf529d3cad30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" Oct 30 06:45:00.440259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440205 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5bt\" (UniqueName: \"kubernetes.io/projected/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-kube-api-access-sb5bt\") pod \"maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7\" (UID: \"12f95efa-c8f1-4e3a-a5db-cf529d3cad30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" Oct 30 06:45:00.440345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440288 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bmc5\" (UniqueName: \"kubernetes.io/projected/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-kube-api-access-5bmc5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq\" (UID: \"1f71c0cf-a14f-4a6b-9067-9d60084a31c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" Oct 30 06:45:00.440416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440398 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x685\" (UniqueName: \"kubernetes.io/projected/fc210264-c902-4f37-b47c-b25ce36a6482-kube-api-access-7x685\") pod \"maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8\" (UID: \"fc210264-c902-4f37-b47c-b25ce36a6482\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" Oct 30 06:45:00.440487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440432 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq\" (UID: \"1f71c0cf-a14f-4a6b-9067-9d60084a31c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" Oct 30 06:45:00.440547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440505 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f0f671-aaa1-4a3f-a267-19d668b62f80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs\" (UID: \"62f0f671-aaa1-4a3f-a267-19d668b62f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" Oct 30 06:45:00.440601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440549 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe6534e6-dec1-42b8-b780-83abc658be3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn\" (UID: \"fe6534e6-dec1-42b8-b780-83abc658be3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" Oct 30 06:45:00.440667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440609 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvm6v\" (UniqueName: \"kubernetes.io/projected/2ced181d-88b9-407f-8663-01b2f28072c2-kube-api-access-vvm6v\") pod \"maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf\" (UID: \"2ced181d-88b9-407f-8663-01b2f28072c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" Oct 30 06:45:00.440667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440650 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ced181d-88b9-407f-8663-01b2f28072c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf\" (UID: \"2ced181d-88b9-407f-8663-01b2f28072c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" Oct 30 06:45:00.440784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440686 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq\" (UID: \"50b32463-c8ed-417f-8718-c5ab4fd2f1b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" Oct 30 06:45:00.440784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440716 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2flf6\" (UniqueName: \"kubernetes.io/projected/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-kube-api-access-2flf6\") pod \"maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq\" (UID: \"50b32463-c8ed-417f-8718-c5ab4fd2f1b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" Oct 30 06:45:00.440784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440769 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p7k5\" (UniqueName: \"kubernetes.io/projected/72dc0962-b2a8-479e-986a-e36ebdfc5e75-kube-api-access-2p7k5\") pod \"maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk\" (UID: \"72dc0962-b2a8-479e-986a-e36ebdfc5e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" Oct 30 06:45:00.440951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440843 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72dc0962-b2a8-479e-986a-e36ebdfc5e75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk\" (UID: \"72dc0962-b2a8-479e-986a-e36ebdfc5e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" Oct 30 06:45:00.440951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440882 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc210264-c902-4f37-b47c-b25ce36a6482-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8\" (UID: \"fc210264-c902-4f37-b47c-b25ce36a6482\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" Oct 30 06:45:00.441063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.440964 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghtpg\" (UniqueName: \"kubernetes.io/projected/fe6534e6-dec1-42b8-b780-83abc658be3e-kube-api-access-ghtpg\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn\" (UID: \"fe6534e6-dec1-42b8-b780-83abc658be3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" Oct 30 06:45:00.441822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.441797 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f0f671-aaa1-4a3f-a267-19d668b62f80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs\" (UID: \"62f0f671-aaa1-4a3f-a267-19d668b62f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" Oct 30 06:45:00.442226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.442204 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq\" (UID: \"50b32463-c8ed-417f-8718-c5ab4fd2f1b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" Oct 30 06:45:00.442581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.442563 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ced181d-88b9-407f-8663-01b2f28072c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf\" (UID: \"2ced181d-88b9-407f-8663-01b2f28072c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" Oct 30 06:45:00.445904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.445886 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x"] Oct 30 06:45:00.446028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.445980 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" Oct 30 06:45:00.456203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.456175 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5"] Oct 30 06:45:00.462677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.462653 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h"] Oct 30 06:45:00.467893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.467865 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjftg\" (UniqueName: \"kubernetes.io/projected/62f0f671-aaa1-4a3f-a267-19d668b62f80-kube-api-access-xjftg\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs\" (UID: \"62f0f671-aaa1-4a3f-a267-19d668b62f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" Oct 30 06:45:00.469084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.469062 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2flf6\" (UniqueName: \"kubernetes.io/projected/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-kube-api-access-2flf6\") pod \"maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq\" (UID: \"50b32463-c8ed-417f-8718-c5ab4fd2f1b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" Oct 30 06:45:00.469380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.469358 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvm6v\" (UniqueName: \"kubernetes.io/projected/2ced181d-88b9-407f-8663-01b2f28072c2-kube-api-access-vvm6v\") pod \"maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf\" (UID: \"2ced181d-88b9-407f-8663-01b2f28072c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" Oct 30 06:45:00.481455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.481421 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl"] Oct 30 06:45:00.539145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.539103 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" Oct 30 06:45:00.543062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.542896 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72dc0962-b2a8-479e-986a-e36ebdfc5e75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk\" (UID: \"72dc0962-b2a8-479e-986a-e36ebdfc5e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" Oct 30 06:45:00.543062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.542937 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc210264-c902-4f37-b47c-b25ce36a6482-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8\" (UID: \"fc210264-c902-4f37-b47c-b25ce36a6482\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" Oct 30 06:45:00.543062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.542991 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghtpg\" (UniqueName: \"kubernetes.io/projected/fe6534e6-dec1-42b8-b780-83abc658be3e-kube-api-access-ghtpg\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn\" (UID: \"fe6534e6-dec1-42b8-b780-83abc658be3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" Oct 30 06:45:00.543062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543021 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/557b0e7c-2d56-49fc-913b-a4da8206101a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl\" (UID: \"557b0e7c-2d56-49fc-913b-a4da8206101a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" Oct 30 06:45:00.543062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543049 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpc84\" (UniqueName: \"kubernetes.io/projected/0d40aa92-6ad3-4340-807b-b4e4796baa3d-kube-api-access-hpc84\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw\" (UID: \"0d40aa92-6ad3-4340-807b-b4e4796baa3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" Oct 30 06:45:00.543420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543189 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvs8\" (UniqueName: \"kubernetes.io/projected/557b0e7c-2d56-49fc-913b-a4da8206101a-kube-api-access-6lvs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl\" (UID: \"557b0e7c-2d56-49fc-913b-a4da8206101a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" Oct 30 06:45:00.543420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543222 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7\" (UID: \"12f95efa-c8f1-4e3a-a5db-cf529d3cad30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" Oct 30 06:45:00.543420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543255 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5bt\" (UniqueName: \"kubernetes.io/projected/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-kube-api-access-sb5bt\") pod \"maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7\" (UID: \"12f95efa-c8f1-4e3a-a5db-cf529d3cad30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" Oct 30 06:45:00.543420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543293 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bmc5\" (UniqueName: \"kubernetes.io/projected/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-kube-api-access-5bmc5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq\" (UID: \"1f71c0cf-a14f-4a6b-9067-9d60084a31c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" Oct 30 06:45:00.543420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543336 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30471359-76f9-4579-be9a-4dd7d3af77a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x\" (UID: \"30471359-76f9-4579-be9a-4dd7d3af77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" Oct 30 06:45:00.543420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543367 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzt7\" (UniqueName: \"kubernetes.io/projected/bb380de3-228b-448b-bc3b-848c1b638b11-kube-api-access-xrzt7\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5\" (UID: \"bb380de3-228b-448b-bc3b-848c1b638b11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" Oct 30 06:45:00.543768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543436 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x685\" (UniqueName: \"kubernetes.io/projected/fc210264-c902-4f37-b47c-b25ce36a6482-kube-api-access-7x685\") pod \"maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8\" (UID: \"fc210264-c902-4f37-b47c-b25ce36a6482\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" Oct 30 06:45:00.543768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543462 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq\" (UID: \"1f71c0cf-a14f-4a6b-9067-9d60084a31c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" Oct 30 06:45:00.543768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543491 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d40aa92-6ad3-4340-807b-b4e4796baa3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw\" (UID: \"0d40aa92-6ad3-4340-807b-b4e4796baa3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" Oct 30 06:45:00.543768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543562 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzpbn\" (UniqueName: \"kubernetes.io/projected/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-kube-api-access-nzpbn\") pod \"maas-api-key-cleanup-29889045-st79h\" (UID: \"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" Oct 30 06:45:00.543768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543611 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889045-st79h\" (UID: \"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" Oct 30 06:45:00.543768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543641 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe6534e6-dec1-42b8-b780-83abc658be3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn\" (UID: \"fe6534e6-dec1-42b8-b780-83abc658be3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" Oct 30 06:45:00.543768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543666 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb380de3-228b-448b-bc3b-848c1b638b11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5\" (UID: \"bb380de3-228b-448b-bc3b-848c1b638b11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" Oct 30 06:45:00.544171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543794 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72dc0962-b2a8-479e-986a-e36ebdfc5e75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk\" (UID: \"72dc0962-b2a8-479e-986a-e36ebdfc5e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" Oct 30 06:45:00.544171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.543836 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc210264-c902-4f37-b47c-b25ce36a6482-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8\" (UID: \"fc210264-c902-4f37-b47c-b25ce36a6482\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" Oct 30 06:45:00.544294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.544168 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74b178e-df08-4063-acdd-a0977b741211-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt\" (UID: \"f74b178e-df08-4063-acdd-a0977b741211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" Oct 30 06:45:00.544294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.544229 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hbht\" (UniqueName: \"kubernetes.io/projected/30471359-76f9-4579-be9a-4dd7d3af77a0-kube-api-access-2hbht\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x\" (UID: \"30471359-76f9-4579-be9a-4dd7d3af77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" Oct 30 06:45:00.544294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.544279 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p7k5\" (UniqueName: \"kubernetes.io/projected/72dc0962-b2a8-479e-986a-e36ebdfc5e75-kube-api-access-2p7k5\") pod \"maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk\" (UID: \"72dc0962-b2a8-479e-986a-e36ebdfc5e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" Oct 30 06:45:00.544522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.544342 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkmpr\" (UniqueName: \"kubernetes.io/projected/f74b178e-df08-4063-acdd-a0977b741211-kube-api-access-lkmpr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt\" (UID: \"f74b178e-df08-4063-acdd-a0977b741211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" Oct 30 06:45:00.544522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.544453 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq\" (UID: \"1f71c0cf-a14f-4a6b-9067-9d60084a31c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" Oct 30 06:45:00.544522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.544453 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe6534e6-dec1-42b8-b780-83abc658be3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn\" (UID: \"fe6534e6-dec1-42b8-b780-83abc658be3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" Oct 30 06:45:00.544795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.544775 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7\" (UID: \"12f95efa-c8f1-4e3a-a5db-cf529d3cad30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" Oct 30 06:45:00.546121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.546094 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" Oct 30 06:45:00.559361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.559282 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" Oct 30 06:45:00.569547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.569517 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5bt\" (UniqueName: \"kubernetes.io/projected/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-kube-api-access-sb5bt\") pod \"maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7\" (UID: \"12f95efa-c8f1-4e3a-a5db-cf529d3cad30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" Oct 30 06:45:00.570039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.569999 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p7k5\" (UniqueName: \"kubernetes.io/projected/72dc0962-b2a8-479e-986a-e36ebdfc5e75-kube-api-access-2p7k5\") pod \"maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk\" (UID: \"72dc0962-b2a8-479e-986a-e36ebdfc5e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" Oct 30 06:45:00.571159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.571132 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x685\" (UniqueName: \"kubernetes.io/projected/fc210264-c902-4f37-b47c-b25ce36a6482-kube-api-access-7x685\") pod \"maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8\" (UID: \"fc210264-c902-4f37-b47c-b25ce36a6482\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" Oct 30 06:45:00.572542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.572512 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bmc5\" (UniqueName: \"kubernetes.io/projected/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-kube-api-access-5bmc5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq\" (UID: \"1f71c0cf-a14f-4a6b-9067-9d60084a31c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" Oct 30 06:45:00.572835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.572813 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghtpg\" (UniqueName: \"kubernetes.io/projected/fe6534e6-dec1-42b8-b780-83abc658be3e-kube-api-access-ghtpg\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn\" (UID: \"fe6534e6-dec1-42b8-b780-83abc658be3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" Oct 30 06:45:00.575685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.575654 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" Oct 30 06:45:00.594545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.594508 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" Oct 30 06:45:00.634804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.634703 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" Oct 30 06:45:00.641250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.641176 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" Oct 30 06:45:00.666145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.664776 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" Oct 30 06:45:00.666145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.665654 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvs8\" (UniqueName: \"kubernetes.io/projected/557b0e7c-2d56-49fc-913b-a4da8206101a-kube-api-access-6lvs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl\" (UID: \"557b0e7c-2d56-49fc-913b-a4da8206101a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" Oct 30 06:45:00.666145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.665738 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30471359-76f9-4579-be9a-4dd7d3af77a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x\" (UID: \"30471359-76f9-4579-be9a-4dd7d3af77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" Oct 30 06:45:00.666145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.665773 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzt7\" (UniqueName: \"kubernetes.io/projected/bb380de3-228b-448b-bc3b-848c1b638b11-kube-api-access-xrzt7\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5\" (UID: \"bb380de3-228b-448b-bc3b-848c1b638b11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" Oct 30 06:45:00.666145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.665852 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d40aa92-6ad3-4340-807b-b4e4796baa3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw\" (UID: \"0d40aa92-6ad3-4340-807b-b4e4796baa3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.668178 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30471359-76f9-4579-be9a-4dd7d3af77a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x\" (UID: \"30471359-76f9-4579-be9a-4dd7d3af77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.668331 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzpbn\" (UniqueName: \"kubernetes.io/projected/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-kube-api-access-nzpbn\") pod \"maas-api-key-cleanup-29889045-st79h\" (UID: \"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.668388 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889045-st79h\" (UID: \"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.668422 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb380de3-228b-448b-bc3b-848c1b638b11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5\" (UID: \"bb380de3-228b-448b-bc3b-848c1b638b11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.668496 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74b178e-df08-4063-acdd-a0977b741211-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt\" (UID: \"f74b178e-df08-4063-acdd-a0977b741211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.668551 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hbht\" (UniqueName: \"kubernetes.io/projected/30471359-76f9-4579-be9a-4dd7d3af77a0-kube-api-access-2hbht\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x\" (UID: \"30471359-76f9-4579-be9a-4dd7d3af77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.668618 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkmpr\" (UniqueName: \"kubernetes.io/projected/f74b178e-df08-4063-acdd-a0977b741211-kube-api-access-lkmpr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt\" (UID: \"f74b178e-df08-4063-acdd-a0977b741211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.668774 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/557b0e7c-2d56-49fc-913b-a4da8206101a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl\" (UID: \"557b0e7c-2d56-49fc-913b-a4da8206101a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.668825 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpc84\" (UniqueName: \"kubernetes.io/projected/0d40aa92-6ad3-4340-807b-b4e4796baa3d-kube-api-access-hpc84\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw\" (UID: \"0d40aa92-6ad3-4340-807b-b4e4796baa3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.669970 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb380de3-228b-448b-bc3b-848c1b638b11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5\" (UID: \"bb380de3-228b-448b-bc3b-848c1b638b11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.669987 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889045-st79h\" (UID: \"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.670592 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74b178e-df08-4063-acdd-a0977b741211-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt\" (UID: \"f74b178e-df08-4063-acdd-a0977b741211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.670691 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d40aa92-6ad3-4340-807b-b4e4796baa3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw\" (UID: \"0d40aa92-6ad3-4340-807b-b4e4796baa3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.671781 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/557b0e7c-2d56-49fc-913b-a4da8206101a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl\" (UID: \"557b0e7c-2d56-49fc-913b-a4da8206101a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" Oct 30 06:45:00.697551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.695558 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" Oct 30 06:45:00.711589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.711522 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzpbn\" (UniqueName: \"kubernetes.io/projected/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-kube-api-access-nzpbn\") pod \"maas-api-key-cleanup-29889045-st79h\" (UID: \"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" Oct 30 06:45:00.715410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.715376 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkmpr\" (UniqueName: \"kubernetes.io/projected/f74b178e-df08-4063-acdd-a0977b741211-kube-api-access-lkmpr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt\" (UID: \"f74b178e-df08-4063-acdd-a0977b741211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" Oct 30 06:45:00.722340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.721588 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzt7\" (UniqueName: \"kubernetes.io/projected/bb380de3-228b-448b-bc3b-848c1b638b11-kube-api-access-xrzt7\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5\" (UID: \"bb380de3-228b-448b-bc3b-848c1b638b11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" Oct 30 06:45:00.722340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.721928 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpc84\" (UniqueName: \"kubernetes.io/projected/0d40aa92-6ad3-4340-807b-b4e4796baa3d-kube-api-access-hpc84\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw\" (UID: \"0d40aa92-6ad3-4340-807b-b4e4796baa3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" Oct 30 06:45:00.726330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.723874 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hbht\" (UniqueName: \"kubernetes.io/projected/30471359-76f9-4579-be9a-4dd7d3af77a0-kube-api-access-2hbht\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x\" (UID: \"30471359-76f9-4579-be9a-4dd7d3af77a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" Oct 30 06:45:00.726330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.724483 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvs8\" (UniqueName: \"kubernetes.io/projected/557b0e7c-2d56-49fc-913b-a4da8206101a-kube-api-access-6lvs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl\" (UID: \"557b0e7c-2d56-49fc-913b-a4da8206101a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" Oct 30 06:45:00.732194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.732161 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" Oct 30 06:45:00.743457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.742822 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" Oct 30 06:45:00.757757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.756726 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" Oct 30 06:45:00.790712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:00.790683 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" Oct 30 06:45:01.006264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.006005 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" Oct 30 06:45:01.017686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.017652 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" Oct 30 06:45:01.208268 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:45:01.208236 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f0f671_aaa1_4a3f_a267_19d668b62f80.slice/crio-4369daec22e3d669ad298cf8fc85841d6697ab571975c38a2c49b73da5d22ae1 WatchSource:0}: Error finding container 4369daec22e3d669ad298cf8fc85841d6697ab571975c38a2c49b73da5d22ae1: Status 404 returned error can't find the container with id 4369daec22e3d669ad298cf8fc85841d6697ab571975c38a2c49b73da5d22ae1 Oct 30 06:45:01.211329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.211281 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs"] Oct 30 06:45:01.339862 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:45:01.339795 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dfdbc8_cb44_4270_8108_11dddce181b9.slice/crio-6622a1d788d55af10873c80d3b0708c891111b54851cb69001bee2be6cb43a79 WatchSource:0}: Error finding container 6622a1d788d55af10873c80d3b0708c891111b54851cb69001bee2be6cb43a79: Status 404 returned error can't find the container with id 6622a1d788d55af10873c80d3b0708c891111b54851cb69001bee2be6cb43a79 Oct 30 06:45:01.347081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.347042 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw"] Oct 30 06:45:01.389900 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:45:01.389856 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b32463_c8ed_417f_8718_c5ab4fd2f1b7.slice/crio-d4b3729e0f016c016de538e178fed946e5b3dc0fa0e1a0bc059b7c05eed83806 WatchSource:0}: Error finding container d4b3729e0f016c016de538e178fed946e5b3dc0fa0e1a0bc059b7c05eed83806: Status 404 returned error can't find the container with id d4b3729e0f016c016de538e178fed946e5b3dc0fa0e1a0bc059b7c05eed83806 Oct 30 06:45:01.391954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.391927 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq"] Oct 30 06:45:01.470492 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:45:01.470461 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ced181d_88b9_407f_8663_01b2f28072c2.slice/crio-efd32b8d2dea2390c2416df4776ea541be570e790db6e5c0fa8d626e8a20a776 WatchSource:0}: Error finding container efd32b8d2dea2390c2416df4776ea541be570e790db6e5c0fa8d626e8a20a776: Status 404 returned error can't find the container with id efd32b8d2dea2390c2416df4776ea541be570e790db6e5c0fa8d626e8a20a776 Oct 30 06:45:01.473227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.473201 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf"] Oct 30 06:45:01.484878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.484854 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7"] Oct 30 06:45:01.542234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.542208 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x"] Oct 30 06:45:01.551553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.551523 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq"] Oct 30 06:45:01.573973 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:45:01.573929 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30471359_76f9_4579_be9a_4dd7d3af77a0.slice/crio-52d2df6c3a591ca231df3ba0d50d65b8246e1eddb43bac42612e97a5a9370f32 WatchSource:0}: Error finding container 52d2df6c3a591ca231df3ba0d50d65b8246e1eddb43bac42612e97a5a9370f32: Status 404 returned error can't find the container with id 52d2df6c3a591ca231df3ba0d50d65b8246e1eddb43bac42612e97a5a9370f32 Oct 30 06:45:01.643964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.643939 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8"] Oct 30 06:45:01.645282 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:45:01.645256 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc210264_c902_4f37_b47c_b25ce36a6482.slice/crio-392440bc035e47129e16d5a6df3dbeafa5be718dc1b3cfae41902287c3fb5b52 WatchSource:0}: Error finding container 392440bc035e47129e16d5a6df3dbeafa5be718dc1b3cfae41902287c3fb5b52: Status 404 returned error can't find the container with id 392440bc035e47129e16d5a6df3dbeafa5be718dc1b3cfae41902287c3fb5b52 Oct 30 06:45:01.673077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.670703 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw"] Oct 30 06:45:01.676969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.676940 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl"] Oct 30 06:45:01.683513 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.683488 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk"] Oct 30 06:45:01.693765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.693742 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5"] Oct 30 06:45:01.746541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.746505 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h"] Oct 30 06:45:01.806591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.806560 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt"] Oct 30 06:45:01.819804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:01.814565 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn"] Oct 30 06:45:02.013921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.013835 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" event={"ID":"0d40aa92-6ad3-4340-807b-b4e4796baa3d","Type":"ContainerStarted","Data":"80b6786380f0717b035293dce6212a264f87bb667546a4027e4b70fa14c6d1c6"} Oct 30 06:45:02.015250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.015216 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" event={"ID":"f74b178e-df08-4063-acdd-a0977b741211","Type":"ContainerStarted","Data":"3a67876d6067e8af118e9839b5c8e27eea73cd70d455c0b9cd566886924a98bc"} Oct 30 06:45:02.016654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.016628 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" event={"ID":"fc210264-c902-4f37-b47c-b25ce36a6482","Type":"ContainerStarted","Data":"392440bc035e47129e16d5a6df3dbeafa5be718dc1b3cfae41902287c3fb5b52"} Oct 30 06:45:02.018415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.018355 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" event={"ID":"62f0f671-aaa1-4a3f-a267-19d668b62f80","Type":"ContainerStarted","Data":"4c993bfa183686a3680359570381ddc1841735fdeb33060b982cf70c60ad9c48"} Oct 30 06:45:02.018415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.018406 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" event={"ID":"62f0f671-aaa1-4a3f-a267-19d668b62f80","Type":"ContainerStarted","Data":"4369daec22e3d669ad298cf8fc85841d6697ab571975c38a2c49b73da5d22ae1"} Oct 30 06:45:02.019805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.019779 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" event={"ID":"70dfdbc8-cb44-4270-8108-11dddce181b9","Type":"ContainerStarted","Data":"6622a1d788d55af10873c80d3b0708c891111b54851cb69001bee2be6cb43a79"} Oct 30 06:45:02.021185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.021165 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" event={"ID":"12f95efa-c8f1-4e3a-a5db-cf529d3cad30","Type":"ContainerStarted","Data":"cec3d2655827a53d2b420d62c1c8ef75dc79c47b55efdd7b087d1067989cc34e"} Oct 30 06:45:02.022678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.022651 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" event={"ID":"30471359-76f9-4579-be9a-4dd7d3af77a0","Type":"ContainerStarted","Data":"52d2df6c3a591ca231df3ba0d50d65b8246e1eddb43bac42612e97a5a9370f32"} Oct 30 06:45:02.024070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.024047 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" event={"ID":"50b32463-c8ed-417f-8718-c5ab4fd2f1b7","Type":"ContainerStarted","Data":"d4b3729e0f016c016de538e178fed946e5b3dc0fa0e1a0bc059b7c05eed83806"} Oct 30 06:45:02.025315 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.025275 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" event={"ID":"72dc0962-b2a8-479e-986a-e36ebdfc5e75","Type":"ContainerStarted","Data":"c4595101b6e5be7f61e3e28f14117003d38ea5e9ca939a7a51e91ca0b30bc6d0"} Oct 30 06:45:02.026668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.026645 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" event={"ID":"fe6534e6-dec1-42b8-b780-83abc658be3e","Type":"ContainerStarted","Data":"e65f194f3b79600daa7593c4ac45771a623465bbd92276a63ec6392114847534"} Oct 30 06:45:02.027999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.027968 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" event={"ID":"1f71c0cf-a14f-4a6b-9067-9d60084a31c6","Type":"ContainerStarted","Data":"63663646c54c1d17562d8b08c9738889d773bda2ecc13f58015c57540569bca2"} Oct 30 06:45:02.029230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.029212 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" event={"ID":"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08","Type":"ContainerStarted","Data":"3093bbeb57795fc4e3e34899deff11a4d3559d3e32f266fe0dae834a02508b35"} Oct 30 06:45:02.030475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.030455 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" event={"ID":"557b0e7c-2d56-49fc-913b-a4da8206101a","Type":"ContainerStarted","Data":"799558e1dfbdea739f3de94a7940af18ed8d50553f28e69131ec42601b360436"} Oct 30 06:45:02.031703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.031684 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" event={"ID":"bb380de3-228b-448b-bc3b-848c1b638b11","Type":"ContainerStarted","Data":"e137411adb0e07e68e2a0a6ff6d42f4adf4a879b407bb6b613f9b62c70a4288a"} Oct 30 06:45:02.032960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.032941 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" event={"ID":"2ced181d-88b9-407f-8663-01b2f28072c2","Type":"ContainerStarted","Data":"efd32b8d2dea2390c2416df4776ea541be570e790db6e5c0fa8d626e8a20a776"} Oct 30 06:45:02.071665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:02.071598 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" podStartSLOduration=2.071560828 podStartE2EDuration="2.071560828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:02.06226469 +0000 UTC m=+3116.058221702" watchObservedRunningTime="2026-10-30 06:45:02.071560828 +0000 UTC m=+3116.067517845" Oct 30 06:45:03.046954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.046442 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" event={"ID":"70dfdbc8-cb44-4270-8108-11dddce181b9","Type":"ContainerStarted","Data":"5234aba596e2dfab8cdcce3ab9ef65f0a3ee11a7473866d78d0f5a71bfd9070c"} Oct 30 06:45:03.049545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.049508 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" event={"ID":"12f95efa-c8f1-4e3a-a5db-cf529d3cad30","Type":"ContainerStarted","Data":"ef6d169c01fdf452a65cbea706061be09d06ca51e910bdfa8457a8997490685c"} Oct 30 06:45:03.057152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.051489 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" event={"ID":"30471359-76f9-4579-be9a-4dd7d3af77a0","Type":"ContainerStarted","Data":"0427168e24dc8d0700dad9cff2aef4f95c6a909dfed73200068e96b50e4f14bf"} Oct 30 06:45:03.061449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.061418 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" event={"ID":"50b32463-c8ed-417f-8718-c5ab4fd2f1b7","Type":"ContainerStarted","Data":"f08913143d07aea71bcd588691c347fc4f5706db6ea3cb0dc7e6a2273ec2488c"} Oct 30 06:45:03.065010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.064963 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" event={"ID":"72dc0962-b2a8-479e-986a-e36ebdfc5e75","Type":"ContainerStarted","Data":"193aebab975df10e4c0ef65ef503c488b58866de234c4515c0c8777ca411de58"} Oct 30 06:45:03.067771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.067735 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" event={"ID":"1f71c0cf-a14f-4a6b-9067-9d60084a31c6","Type":"ContainerStarted","Data":"b7b9bb3e07f8f69cb828e63ae5334eb640c6a9e88e2e93a722fe4d178df2c81f"} Oct 30 06:45:03.070211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.070182 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" event={"ID":"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08","Type":"ContainerStarted","Data":"7094c8936fa4b3a849c090ba330049d32d551a180264e237f7362b2abad2e6b6"} Oct 30 06:45:03.072267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.072238 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" event={"ID":"557b0e7c-2d56-49fc-913b-a4da8206101a","Type":"ContainerStarted","Data":"118258841ba1e49957b6025ec4c74ca9934103695a635ae5adc936d9310504ea"} Oct 30 06:45:03.074419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.074394 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" event={"ID":"bb380de3-228b-448b-bc3b-848c1b638b11","Type":"ContainerStarted","Data":"980591b4922303e4166c60fe8079fa2ec4143552fc19361d95c0be537ed7e9ae"} Oct 30 06:45:03.076380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.076356 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" event={"ID":"2ced181d-88b9-407f-8663-01b2f28072c2","Type":"ContainerStarted","Data":"0ef4f74be3b4ef3e07c238b78499ce9dbc6fafeec2dcaa8a9d3dbabb13d676c3"} Oct 30 06:45:03.078051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.078030 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" event={"ID":"0d40aa92-6ad3-4340-807b-b4e4796baa3d","Type":"ContainerStarted","Data":"b98cacc880911c85bc719cf31ac0f6231098537162baa7d0e41830b4834cf425"} Oct 30 06:45:03.079923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.079892 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" event={"ID":"f74b178e-df08-4063-acdd-a0977b741211","Type":"ContainerStarted","Data":"1f7928f368743e81f7e8fdebc2aace13079590dbeaa9f1481abfbd0d14aa2361"} Oct 30 06:45:03.081691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.081670 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" event={"ID":"fc210264-c902-4f37-b47c-b25ce36a6482","Type":"ContainerStarted","Data":"737a16bfc548592e48490095d9eaf3a8ca8f03248fc97ce72c06bf7724830c02"} Oct 30 06:45:03.083419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.083395 5001 generic.go:356] "Generic (PLEG): container finished" podID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerID="4c993bfa183686a3680359570381ddc1841735fdeb33060b982cf70c60ad9c48" exitCode=60 Oct 30 06:45:03.083516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.083455 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" event={"ID":"62f0f671-aaa1-4a3f-a267-19d668b62f80","Type":"ContainerDied","Data":"4c993bfa183686a3680359570381ddc1841735fdeb33060b982cf70c60ad9c48"} Oct 30 06:45:03.083757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.083738 5001 scope.go:122] "RemoveContainer" containerID="4c993bfa183686a3680359570381ddc1841735fdeb33060b982cf70c60ad9c48" Oct 30 06:45:03.094127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.094066 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" podStartSLOduration=3.094048478 podStartE2EDuration="3.094048478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.087444007 +0000 UTC m=+3117.083401018" watchObservedRunningTime="2026-10-30 06:45:03.094048478 +0000 UTC m=+3117.090005489" Oct 30 06:45:03.138297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.138233 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" podStartSLOduration=3.138216005 podStartE2EDuration="3.138216005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.132974134 +0000 UTC m=+3117.128931145" watchObservedRunningTime="2026-10-30 06:45:03.138216005 +0000 UTC m=+3117.134173017" Oct 30 06:45:03.242167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.242098 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" podStartSLOduration=3.242075239 podStartE2EDuration="3.242075239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.203650555 +0000 UTC m=+3117.199607566" watchObservedRunningTime="2026-10-30 06:45:03.242075239 +0000 UTC m=+3117.238032249" Oct 30 06:45:03.293056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.293001 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" podStartSLOduration=3.292986068 podStartE2EDuration="3.292986068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.291165093 +0000 UTC m=+3117.287122104" watchObservedRunningTime="2026-10-30 06:45:03.292986068 +0000 UTC m=+3117.288943070" Oct 30 06:45:03.336648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.336588 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" podStartSLOduration=3.336573152 podStartE2EDuration="3.336573152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.331834706 +0000 UTC m=+3117.327791716" watchObservedRunningTime="2026-10-30 06:45:03.336573152 +0000 UTC m=+3117.332530161" Oct 30 06:45:03.422965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.422897 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" podStartSLOduration=3.422878476 podStartE2EDuration="3.422878476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.417637083 +0000 UTC m=+3117.413594093" watchObservedRunningTime="2026-10-30 06:45:03.422878476 +0000 UTC m=+3117.418835487" Oct 30 06:45:03.495122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.495055 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" podStartSLOduration=3.495036071 podStartE2EDuration="3.495036071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.492140883 +0000 UTC m=+3117.488097896" watchObservedRunningTime="2026-10-30 06:45:03.495036071 +0000 UTC m=+3117.490993083" Oct 30 06:45:03.569534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.569478 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" podStartSLOduration=3.56946008 podStartE2EDuration="3.56946008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.565903857 +0000 UTC m=+3117.561860868" watchObservedRunningTime="2026-10-30 06:45:03.56946008 +0000 UTC m=+3117.565417090" Oct 30 06:45:03.618687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.618636 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" podStartSLOduration=3.618620939 podStartE2EDuration="3.618620939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.614538169 +0000 UTC m=+3117.610495190" watchObservedRunningTime="2026-10-30 06:45:03.618620939 +0000 UTC m=+3117.614577948" Oct 30 06:45:03.658968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.658610 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" podStartSLOduration=3.658590495 podStartE2EDuration="3.658590495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.655984214 +0000 UTC m=+3117.651941234" watchObservedRunningTime="2026-10-30 06:45:03.658590495 +0000 UTC m=+3117.654547505" Oct 30 06:45:03.702258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.702199 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" podStartSLOduration=3.702181668 podStartE2EDuration="3.702181668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.696670894 +0000 UTC m=+3117.692627904" watchObservedRunningTime="2026-10-30 06:45:03.702181668 +0000 UTC m=+3117.698138677" Oct 30 06:45:03.744789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.744728 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" podStartSLOduration=3.744713494 podStartE2EDuration="3.744713494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.742291911 +0000 UTC m=+3117.738248922" watchObservedRunningTime="2026-10-30 06:45:03.744713494 +0000 UTC m=+3117.740670500" Oct 30 06:45:03.787679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:03.787623 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" podStartSLOduration=3.787607723 podStartE2EDuration="3.787607723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.783718557 +0000 UTC m=+3117.779675567" watchObservedRunningTime="2026-10-30 06:45:03.787607723 +0000 UTC m=+3117.783564776" Oct 30 06:45:04.102777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.102682 5001 generic.go:356] "Generic (PLEG): container finished" podID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerID="0427168e24dc8d0700dad9cff2aef4f95c6a909dfed73200068e96b50e4f14bf" exitCode=60 Oct 30 06:45:04.103215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.102772 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" event={"ID":"30471359-76f9-4579-be9a-4dd7d3af77a0","Type":"ContainerDied","Data":"0427168e24dc8d0700dad9cff2aef4f95c6a909dfed73200068e96b50e4f14bf"} Oct 30 06:45:04.103215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.103136 5001 scope.go:122] "RemoveContainer" containerID="0427168e24dc8d0700dad9cff2aef4f95c6a909dfed73200068e96b50e4f14bf" Oct 30 06:45:04.105694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.105630 5001 generic.go:356] "Generic (PLEG): container finished" podID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerID="f08913143d07aea71bcd588691c347fc4f5706db6ea3cb0dc7e6a2273ec2488c" exitCode=60 Oct 30 06:45:04.105846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.105749 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" event={"ID":"50b32463-c8ed-417f-8718-c5ab4fd2f1b7","Type":"ContainerDied","Data":"f08913143d07aea71bcd588691c347fc4f5706db6ea3cb0dc7e6a2273ec2488c"} Oct 30 06:45:04.106076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.106053 5001 scope.go:122] "RemoveContainer" containerID="f08913143d07aea71bcd588691c347fc4f5706db6ea3cb0dc7e6a2273ec2488c" Oct 30 06:45:04.111723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.108592 5001 generic.go:356] "Generic (PLEG): container finished" podID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerID="193aebab975df10e4c0ef65ef503c488b58866de234c4515c0c8777ca411de58" exitCode=60 Oct 30 06:45:04.111723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.108749 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" event={"ID":"72dc0962-b2a8-479e-986a-e36ebdfc5e75","Type":"ContainerDied","Data":"193aebab975df10e4c0ef65ef503c488b58866de234c4515c0c8777ca411de58"} Oct 30 06:45:04.111723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.109330 5001 scope.go:122] "RemoveContainer" containerID="193aebab975df10e4c0ef65ef503c488b58866de234c4515c0c8777ca411de58" Oct 30 06:45:04.113262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.113232 5001 generic.go:356] "Generic (PLEG): container finished" podID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerID="20f1691c827ee89bce8c06db4ac76804983eb26a47819870b54d897f2d012e9b" exitCode=60 Oct 30 06:45:04.114119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.114099 5001 scope.go:122] "RemoveContainer" containerID="20f1691c827ee89bce8c06db4ac76804983eb26a47819870b54d897f2d012e9b" Oct 30 06:45:04.126432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.126400 5001 generic.go:356] "Generic (PLEG): container finished" podID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerID="b7b9bb3e07f8f69cb828e63ae5334eb640c6a9e88e2e93a722fe4d178df2c81f" exitCode=60 Oct 30 06:45:04.127177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.127152 5001 scope.go:122] "RemoveContainer" containerID="b7b9bb3e07f8f69cb828e63ae5334eb640c6a9e88e2e93a722fe4d178df2c81f" Oct 30 06:45:04.140120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.140085 5001 generic.go:356] "Generic (PLEG): container finished" podID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerID="7094c8936fa4b3a849c090ba330049d32d551a180264e237f7362b2abad2e6b6" exitCode=60 Oct 30 06:45:04.140705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.140679 5001 scope.go:122] "RemoveContainer" containerID="7094c8936fa4b3a849c090ba330049d32d551a180264e237f7362b2abad2e6b6" Oct 30 06:45:04.141487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.141207 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" event={"ID":"fe6534e6-dec1-42b8-b780-83abc658be3e","Type":"ContainerDied","Data":"20f1691c827ee89bce8c06db4ac76804983eb26a47819870b54d897f2d012e9b"} Oct 30 06:45:04.141487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.141268 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" event={"ID":"1f71c0cf-a14f-4a6b-9067-9d60084a31c6","Type":"ContainerDied","Data":"b7b9bb3e07f8f69cb828e63ae5334eb640c6a9e88e2e93a722fe4d178df2c81f"} Oct 30 06:45:04.141487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.141297 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" event={"ID":"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08","Type":"ContainerDied","Data":"7094c8936fa4b3a849c090ba330049d32d551a180264e237f7362b2abad2e6b6"} Oct 30 06:45:04.145243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.145204 5001 generic.go:356] "Generic (PLEG): container finished" podID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerID="118258841ba1e49957b6025ec4c74ca9934103695a635ae5adc936d9310504ea" exitCode=60 Oct 30 06:45:04.145381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.145291 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" event={"ID":"557b0e7c-2d56-49fc-913b-a4da8206101a","Type":"ContainerDied","Data":"118258841ba1e49957b6025ec4c74ca9934103695a635ae5adc936d9310504ea"} Oct 30 06:45:04.145860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.145839 5001 scope.go:122] "RemoveContainer" containerID="118258841ba1e49957b6025ec4c74ca9934103695a635ae5adc936d9310504ea" Oct 30 06:45:04.155998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.155773 5001 generic.go:356] "Generic (PLEG): container finished" podID="bb380de3-228b-448b-bc3b-848c1b638b11" containerID="980591b4922303e4166c60fe8079fa2ec4143552fc19361d95c0be537ed7e9ae" exitCode=60 Oct 30 06:45:04.155998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.155880 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" event={"ID":"bb380de3-228b-448b-bc3b-848c1b638b11","Type":"ContainerDied","Data":"980591b4922303e4166c60fe8079fa2ec4143552fc19361d95c0be537ed7e9ae"} Oct 30 06:45:04.156778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.156757 5001 scope.go:122] "RemoveContainer" containerID="980591b4922303e4166c60fe8079fa2ec4143552fc19361d95c0be537ed7e9ae" Oct 30 06:45:04.158124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.158066 5001 generic.go:356] "Generic (PLEG): container finished" podID="2ced181d-88b9-407f-8663-01b2f28072c2" containerID="0ef4f74be3b4ef3e07c238b78499ce9dbc6fafeec2dcaa8a9d3dbabb13d676c3" exitCode=60 Oct 30 06:45:04.158224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.158145 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" event={"ID":"2ced181d-88b9-407f-8663-01b2f28072c2","Type":"ContainerDied","Data":"0ef4f74be3b4ef3e07c238b78499ce9dbc6fafeec2dcaa8a9d3dbabb13d676c3"} Oct 30 06:45:04.158769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.158752 5001 scope.go:122] "RemoveContainer" containerID="0ef4f74be3b4ef3e07c238b78499ce9dbc6fafeec2dcaa8a9d3dbabb13d676c3" Oct 30 06:45:04.162553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.162477 5001 generic.go:356] "Generic (PLEG): container finished" podID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerID="b98cacc880911c85bc719cf31ac0f6231098537162baa7d0e41830b4834cf425" exitCode=60 Oct 30 06:45:04.162785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.162760 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" event={"ID":"0d40aa92-6ad3-4340-807b-b4e4796baa3d","Type":"ContainerDied","Data":"b98cacc880911c85bc719cf31ac0f6231098537162baa7d0e41830b4834cf425"} Oct 30 06:45:04.163020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.162949 5001 scope.go:122] "RemoveContainer" containerID="b98cacc880911c85bc719cf31ac0f6231098537162baa7d0e41830b4834cf425" Oct 30 06:45:04.165414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.165217 5001 generic.go:356] "Generic (PLEG): container finished" podID="f74b178e-df08-4063-acdd-a0977b741211" containerID="1f7928f368743e81f7e8fdebc2aace13079590dbeaa9f1481abfbd0d14aa2361" exitCode=60 Oct 30 06:45:04.165414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.165285 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" event={"ID":"f74b178e-df08-4063-acdd-a0977b741211","Type":"ContainerDied","Data":"1f7928f368743e81f7e8fdebc2aace13079590dbeaa9f1481abfbd0d14aa2361"} Oct 30 06:45:04.165894 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.165817 5001 scope.go:122] "RemoveContainer" containerID="1f7928f368743e81f7e8fdebc2aace13079590dbeaa9f1481abfbd0d14aa2361" Oct 30 06:45:04.173254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.172924 5001 generic.go:356] "Generic (PLEG): container finished" podID="fc210264-c902-4f37-b47c-b25ce36a6482" containerID="737a16bfc548592e48490095d9eaf3a8ca8f03248fc97ce72c06bf7724830c02" exitCode=60 Oct 30 06:45:04.173254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.173033 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" event={"ID":"fc210264-c902-4f37-b47c-b25ce36a6482","Type":"ContainerDied","Data":"737a16bfc548592e48490095d9eaf3a8ca8f03248fc97ce72c06bf7724830c02"} Oct 30 06:45:04.173788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.173738 5001 scope.go:122] "RemoveContainer" containerID="737a16bfc548592e48490095d9eaf3a8ca8f03248fc97ce72c06bf7724830c02" Oct 30 06:45:04.178549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.177807 5001 generic.go:356] "Generic (PLEG): container finished" podID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerID="5234aba596e2dfab8cdcce3ab9ef65f0a3ee11a7473866d78d0f5a71bfd9070c" exitCode=60 Oct 30 06:45:04.178549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.177976 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" event={"ID":"70dfdbc8-cb44-4270-8108-11dddce181b9","Type":"ContainerDied","Data":"5234aba596e2dfab8cdcce3ab9ef65f0a3ee11a7473866d78d0f5a71bfd9070c"} Oct 30 06:45:04.178549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.178242 5001 scope.go:122] "RemoveContainer" containerID="5234aba596e2dfab8cdcce3ab9ef65f0a3ee11a7473866d78d0f5a71bfd9070c" Oct 30 06:45:04.181556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.181532 5001 generic.go:356] "Generic (PLEG): container finished" podID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerID="ef6d169c01fdf452a65cbea706061be09d06ca51e910bdfa8457a8997490685c" exitCode=60 Oct 30 06:45:04.181663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.181644 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" event={"ID":"12f95efa-c8f1-4e3a-a5db-cf529d3cad30","Type":"ContainerDied","Data":"ef6d169c01fdf452a65cbea706061be09d06ca51e910bdfa8457a8997490685c"} Oct 30 06:45:04.181968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:04.181906 5001 scope.go:122] "RemoveContainer" containerID="ef6d169c01fdf452a65cbea706061be09d06ca51e910bdfa8457a8997490685c" Oct 30 06:45:05.197458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:05.197418 5001 generic.go:356] "Generic (PLEG): container finished" podID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerID="dd44523be1aeb68e1901b4d304511366b8c70aac10bc74aae9b338c63b344f40" exitCode=60 Oct 30 06:45:05.197936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:05.197502 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" event={"ID":"62f0f671-aaa1-4a3f-a267-19d668b62f80","Type":"ContainerDied","Data":"dd44523be1aeb68e1901b4d304511366b8c70aac10bc74aae9b338c63b344f40"} Oct 30 06:45:05.197936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:05.197578 5001 scope.go:122] "RemoveContainer" containerID="4c993bfa183686a3680359570381ddc1841735fdeb33060b982cf70c60ad9c48" Oct 30 06:45:05.197936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:05.197889 5001 scope.go:122] "RemoveContainer" containerID="dd44523be1aeb68e1901b4d304511366b8c70aac10bc74aae9b338c63b344f40" Oct 30 06:45:05.198271 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:05.198243 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs_odh-ai-gateway-infra(62f0f671-aaa1-4a3f-a267-19d668b62f80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" Oct 30 06:45:05.201212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:05.201185 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" event={"ID":"50b32463-c8ed-417f-8718-c5ab4fd2f1b7","Type":"ContainerStarted","Data":"5f08de7564f280c18f5d11d7c312289830bb7d24b71bedbec388df80e73c5672"} Oct 30 06:45:05.203738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:05.203663 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" event={"ID":"72dc0962-b2a8-479e-986a-e36ebdfc5e75","Type":"ContainerStarted","Data":"aa176122c3040e97cb533a1b9e961ee638bb0084050e670cbca43aed6564a027"} Oct 30 06:45:05.205789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:05.205771 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" event={"ID":"1f71c0cf-a14f-4a6b-9067-9d60084a31c6","Type":"ContainerStarted","Data":"54b6c2574201556010c88aa29a741c34619190052fc189a8e06fb5680411d749"} Oct 30 06:45:05.987128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:05.987093 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt"] Oct 30 06:45:05.992514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:05.992490 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.008091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.008039 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"openshift-service-ca.crt\"" Oct 30 06:45:06.008324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.008120 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"kube-root-ca.crt\"" Oct 30 06:45:06.008324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.008127 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"agc-cred-39512e62d644\"" Oct 30 06:45:06.008324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.008163 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"routing-overlay\"" Oct 30 06:45:06.008324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.008252 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-plugins-e2e-extmod-a-88acce\"" Oct 30 06:45:06.008567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.008430 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-e2e--4e48ea35-dockercfg-v22pn\"" Oct 30 06:45:06.011872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.011841 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt"] Oct 30 06:45:06.033026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.032980 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.035057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.035016 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.035057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.035058 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.035263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.035082 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjmj\" (UniqueName: \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-kube-api-access-9gjmj\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.137227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.137193 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.137440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.137364 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.137440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.137389 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.137440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.137405 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjmj\" (UniqueName: \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-kube-api-access-9gjmj\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.138261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.138232 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.138561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.138543 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.140319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.140283 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.163536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.163501 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjmj\" (UniqueName: \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-kube-api-access-9gjmj\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.220914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.220868 5001 generic.go:356] "Generic (PLEG): container finished" podID="fc210264-c902-4f37-b47c-b25ce36a6482" containerID="681d20e0c3b609dade73fab92c5d75896d87ea3a948f0c7ac76e6cc444201855" exitCode=60 Oct 30 06:45:06.221461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.220974 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" event={"ID":"fc210264-c902-4f37-b47c-b25ce36a6482","Type":"ContainerDied","Data":"681d20e0c3b609dade73fab92c5d75896d87ea3a948f0c7ac76e6cc444201855"} Oct 30 06:45:06.221461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.221040 5001 scope.go:122] "RemoveContainer" containerID="737a16bfc548592e48490095d9eaf3a8ca8f03248fc97ce72c06bf7724830c02" Oct 30 06:45:06.221461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.221455 5001 scope.go:122] "RemoveContainer" containerID="681d20e0c3b609dade73fab92c5d75896d87ea3a948f0c7ac76e6cc444201855" Oct 30 06:45:06.221954 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.221927 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8_odh-ai-gateway-infra(fc210264-c902-4f37-b47c-b25ce36a6482)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" Oct 30 06:45:06.223687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.223659 5001 scope.go:122] "RemoveContainer" containerID="dd44523be1aeb68e1901b4d304511366b8c70aac10bc74aae9b338c63b344f40" Oct 30 06:45:06.224075 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.224051 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs_odh-ai-gateway-infra(62f0f671-aaa1-4a3f-a267-19d668b62f80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" Oct 30 06:45:06.225675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.225650 5001 generic.go:356] "Generic (PLEG): container finished" podID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerID="3bb3da9cf0087186f2c2e86607d65514e3569b63d8750a2062e83c5424ae1c7d" exitCode=60 Oct 30 06:45:06.225781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.225748 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" event={"ID":"70dfdbc8-cb44-4270-8108-11dddce181b9","Type":"ContainerDied","Data":"3bb3da9cf0087186f2c2e86607d65514e3569b63d8750a2062e83c5424ae1c7d"} Oct 30 06:45:06.226039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.226016 5001 scope.go:122] "RemoveContainer" containerID="3bb3da9cf0087186f2c2e86607d65514e3569b63d8750a2062e83c5424ae1c7d" Oct 30 06:45:06.226472 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.226442 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw_odh-ai-gateway-infra(70dfdbc8-cb44-4270-8108-11dddce181b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" Oct 30 06:45:06.228603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.228578 5001 generic.go:356] "Generic (PLEG): container finished" podID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerID="d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb" exitCode=60 Oct 30 06:45:06.228729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.228665 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" event={"ID":"12f95efa-c8f1-4e3a-a5db-cf529d3cad30","Type":"ContainerDied","Data":"d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb"} Oct 30 06:45:06.228909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.228893 5001 scope.go:122] "RemoveContainer" containerID="d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb" Oct 30 06:45:06.229243 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.229218 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7_odh-ai-gateway-infra(12f95efa-c8f1-4e3a-a5db-cf529d3cad30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" Oct 30 06:45:06.230751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.230731 5001 generic.go:356] "Generic (PLEG): container finished" podID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerID="9c9841402efe937919b5d873b570c01fccc36b596685cc53ffe016ae4abc025d" exitCode=60 Oct 30 06:45:06.230867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.230814 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" event={"ID":"30471359-76f9-4579-be9a-4dd7d3af77a0","Type":"ContainerDied","Data":"9c9841402efe937919b5d873b570c01fccc36b596685cc53ffe016ae4abc025d"} Oct 30 06:45:06.231142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.231125 5001 scope.go:122] "RemoveContainer" containerID="9c9841402efe937919b5d873b570c01fccc36b596685cc53ffe016ae4abc025d" Oct 30 06:45:06.231513 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.231490 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x_odh-ai-gateway-infra(30471359-76f9-4579-be9a-4dd7d3af77a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" Oct 30 06:45:06.232717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.232695 5001 generic.go:356] "Generic (PLEG): container finished" podID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerID="5f08de7564f280c18f5d11d7c312289830bb7d24b71bedbec388df80e73c5672" exitCode=60 Oct 30 06:45:06.232804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.232788 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" event={"ID":"50b32463-c8ed-417f-8718-c5ab4fd2f1b7","Type":"ContainerDied","Data":"5f08de7564f280c18f5d11d7c312289830bb7d24b71bedbec388df80e73c5672"} Oct 30 06:45:06.232978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.232963 5001 scope.go:122] "RemoveContainer" containerID="5f08de7564f280c18f5d11d7c312289830bb7d24b71bedbec388df80e73c5672" Oct 30 06:45:06.233243 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.233218 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq_odh-ai-gateway-infra(50b32463-c8ed-417f-8718-c5ab4fd2f1b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" Oct 30 06:45:06.234932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.234912 5001 generic.go:356] "Generic (PLEG): container finished" podID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerID="aa176122c3040e97cb533a1b9e961ee638bb0084050e670cbca43aed6564a027" exitCode=60 Oct 30 06:45:06.235030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.234990 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" event={"ID":"72dc0962-b2a8-479e-986a-e36ebdfc5e75","Type":"ContainerDied","Data":"aa176122c3040e97cb533a1b9e961ee638bb0084050e670cbca43aed6564a027"} Oct 30 06:45:06.235225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.235209 5001 scope.go:122] "RemoveContainer" containerID="aa176122c3040e97cb533a1b9e961ee638bb0084050e670cbca43aed6564a027" Oct 30 06:45:06.235751 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.235721 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk_odh-ai-gateway-infra(72dc0962-b2a8-479e-986a-e36ebdfc5e75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" Oct 30 06:45:06.237135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.237113 5001 generic.go:356] "Generic (PLEG): container finished" podID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerID="9bcce567bd8a81a49ed6de1a95cab4063c4a961b7eb82b8d497592cbfdc57638" exitCode=60 Oct 30 06:45:06.237233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.237184 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" event={"ID":"fe6534e6-dec1-42b8-b780-83abc658be3e","Type":"ContainerDied","Data":"9bcce567bd8a81a49ed6de1a95cab4063c4a961b7eb82b8d497592cbfdc57638"} Oct 30 06:45:06.237526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.237509 5001 scope.go:122] "RemoveContainer" containerID="9bcce567bd8a81a49ed6de1a95cab4063c4a961b7eb82b8d497592cbfdc57638" Oct 30 06:45:06.238176 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.238153 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn_odh-ai-gateway-infra(fe6534e6-dec1-42b8-b780-83abc658be3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" Oct 30 06:45:06.239544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.239526 5001 generic.go:356] "Generic (PLEG): container finished" podID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerID="54b6c2574201556010c88aa29a741c34619190052fc189a8e06fb5680411d749" exitCode=60 Oct 30 06:45:06.239667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.239607 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" event={"ID":"1f71c0cf-a14f-4a6b-9067-9d60084a31c6","Type":"ContainerDied","Data":"54b6c2574201556010c88aa29a741c34619190052fc189a8e06fb5680411d749"} Oct 30 06:45:06.239836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.239813 5001 scope.go:122] "RemoveContainer" containerID="54b6c2574201556010c88aa29a741c34619190052fc189a8e06fb5680411d749" Oct 30 06:45:06.240115 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.240087 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq_odh-ai-gateway-infra(1f71c0cf-a14f-4a6b-9067-9d60084a31c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" Oct 30 06:45:06.241971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.241930 5001 generic.go:356] "Generic (PLEG): container finished" podID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerID="03a020a466e39b4ad78e16a44a6b842c51ce78d6a4931452c4bc81a3123e5dcf" exitCode=60 Oct 30 06:45:06.242108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.242089 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" event={"ID":"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08","Type":"ContainerDied","Data":"03a020a466e39b4ad78e16a44a6b842c51ce78d6a4931452c4bc81a3123e5dcf"} Oct 30 06:45:06.242268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.242245 5001 scope.go:122] "RemoveContainer" containerID="03a020a466e39b4ad78e16a44a6b842c51ce78d6a4931452c4bc81a3123e5dcf" Oct 30 06:45:06.242588 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.242565 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889045-st79h_odh-ai-gateway-infra(c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" Oct 30 06:45:06.244483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.244462 5001 generic.go:356] "Generic (PLEG): container finished" podID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerID="987cc769b5fbcc3095e6e8300e4dd9e65f67bf10054fd13123946fc809fdfe55" exitCode=60 Oct 30 06:45:06.244595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.244539 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" event={"ID":"557b0e7c-2d56-49fc-913b-a4da8206101a","Type":"ContainerDied","Data":"987cc769b5fbcc3095e6e8300e4dd9e65f67bf10054fd13123946fc809fdfe55"} Oct 30 06:45:06.244770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.244754 5001 scope.go:122] "RemoveContainer" containerID="987cc769b5fbcc3095e6e8300e4dd9e65f67bf10054fd13123946fc809fdfe55" Oct 30 06:45:06.245026 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.245001 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl_odh-ai-gateway-infra(557b0e7c-2d56-49fc-913b-a4da8206101a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" Oct 30 06:45:06.246578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.246555 5001 generic.go:356] "Generic (PLEG): container finished" podID="bb380de3-228b-448b-bc3b-848c1b638b11" containerID="84d4f51e86c442d2f1fe2dc7f448cbaf95a464740395d59d17049f670322c228" exitCode=60 Oct 30 06:45:06.246686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.246628 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" event={"ID":"bb380de3-228b-448b-bc3b-848c1b638b11","Type":"ContainerDied","Data":"84d4f51e86c442d2f1fe2dc7f448cbaf95a464740395d59d17049f670322c228"} Oct 30 06:45:06.246902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.246884 5001 scope.go:122] "RemoveContainer" containerID="84d4f51e86c442d2f1fe2dc7f448cbaf95a464740395d59d17049f670322c228" Oct 30 06:45:06.247127 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.247105 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5_odh-ai-gateway-infra(bb380de3-228b-448b-bc3b-848c1b638b11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" Oct 30 06:45:06.248784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.248759 5001 generic.go:356] "Generic (PLEG): container finished" podID="2ced181d-88b9-407f-8663-01b2f28072c2" containerID="822b98090dcf0e4abd4459a34ac3ee68f3bdd8677399ec4d7996c880f462dd48" exitCode=60 Oct 30 06:45:06.248933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.248918 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" event={"ID":"2ced181d-88b9-407f-8663-01b2f28072c2","Type":"ContainerDied","Data":"822b98090dcf0e4abd4459a34ac3ee68f3bdd8677399ec4d7996c880f462dd48"} Oct 30 06:45:06.249065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.249049 5001 scope.go:122] "RemoveContainer" containerID="822b98090dcf0e4abd4459a34ac3ee68f3bdd8677399ec4d7996c880f462dd48" Oct 30 06:45:06.249287 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.249266 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf_odh-ai-gateway-infra(2ced181d-88b9-407f-8663-01b2f28072c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" Oct 30 06:45:06.250967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.250948 5001 generic.go:356] "Generic (PLEG): container finished" podID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerID="8a3e7897ec6ef7c1323bbe53e4b4318f76eea7cca854d21cdfc51932ceef87af" exitCode=60 Oct 30 06:45:06.251082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.251025 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" event={"ID":"0d40aa92-6ad3-4340-807b-b4e4796baa3d","Type":"ContainerDied","Data":"8a3e7897ec6ef7c1323bbe53e4b4318f76eea7cca854d21cdfc51932ceef87af"} Oct 30 06:45:06.251213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.251200 5001 scope.go:122] "RemoveContainer" containerID="8a3e7897ec6ef7c1323bbe53e4b4318f76eea7cca854d21cdfc51932ceef87af" Oct 30 06:45:06.251502 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.251480 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw_odh-ai-gateway-infra(0d40aa92-6ad3-4340-807b-b4e4796baa3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" Oct 30 06:45:06.253083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.253063 5001 generic.go:356] "Generic (PLEG): container finished" podID="f74b178e-df08-4063-acdd-a0977b741211" containerID="46a9a2d2436f13346046b1ded9780d3aa42d5b1b2ebd77b4a85b10c7fdbb1f0d" exitCode=60 Oct 30 06:45:06.253175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.253097 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" event={"ID":"f74b178e-df08-4063-acdd-a0977b741211","Type":"ContainerDied","Data":"46a9a2d2436f13346046b1ded9780d3aa42d5b1b2ebd77b4a85b10c7fdbb1f0d"} Oct 30 06:45:06.253395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.253378 5001 scope.go:122] "RemoveContainer" containerID="46a9a2d2436f13346046b1ded9780d3aa42d5b1b2ebd77b4a85b10c7fdbb1f0d" Oct 30 06:45:06.253657 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:06.253637 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt_odh-ai-gateway-infra(f74b178e-df08-4063-acdd-a0977b741211)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" podUID="f74b178e-df08-4063-acdd-a0977b741211" Oct 30 06:45:06.537817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.537789 5001 scope.go:122] "RemoveContainer" containerID="5234aba596e2dfab8cdcce3ab9ef65f0a3ee11a7473866d78d0f5a71bfd9070c" Oct 30 06:45:06.555423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.555218 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-e2e--4e48ea35-dockercfg-v22pn\"" Oct 30 06:45:06.560480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.560453 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:06.581830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.578563 5001 scope.go:122] "RemoveContainer" containerID="ef6d169c01fdf452a65cbea706061be09d06ca51e910bdfa8457a8997490685c" Oct 30 06:45:06.634022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.633993 5001 scope.go:122] "RemoveContainer" containerID="0427168e24dc8d0700dad9cff2aef4f95c6a909dfed73200068e96b50e4f14bf" Oct 30 06:45:06.680121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.669501 5001 scope.go:122] "RemoveContainer" containerID="f08913143d07aea71bcd588691c347fc4f5706db6ea3cb0dc7e6a2273ec2488c" Oct 30 06:45:06.701464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.700192 5001 scope.go:122] "RemoveContainer" containerID="193aebab975df10e4c0ef65ef503c488b58866de234c4515c0c8777ca411de58" Oct 30 06:45:06.721294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.721257 5001 scope.go:122] "RemoveContainer" containerID="20f1691c827ee89bce8c06db4ac76804983eb26a47819870b54d897f2d012e9b" Oct 30 06:45:06.740137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.740111 5001 scope.go:122] "RemoveContainer" containerID="b7b9bb3e07f8f69cb828e63ae5334eb640c6a9e88e2e93a722fe4d178df2c81f" Oct 30 06:45:06.766821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:06.766658 5001 scope.go:122] "RemoveContainer" containerID="7094c8936fa4b3a849c090ba330049d32d551a180264e237f7362b2abad2e6b6" Oct 30 06:45:07.084333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.076119 5001 scope.go:122] "RemoveContainer" containerID="118258841ba1e49957b6025ec4c74ca9934103695a635ae5adc936d9310504ea" Oct 30 06:45:07.205150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.205119 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9"] Oct 30 06:45:07.210368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.206341 5001 scope.go:122] "RemoveContainer" containerID="980591b4922303e4166c60fe8079fa2ec4143552fc19361d95c0be537ed7e9ae" Oct 30 06:45:07.210368 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:45:07.209428 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda346576f_1151_4520_9dd7_dec9739f2032.slice/crio-544d4bff588275706104eab45cf608478f9c86e8706c1f47ed07547368daf0d6 WatchSource:0}: Error finding container 544d4bff588275706104eab45cf608478f9c86e8706c1f47ed07547368daf0d6: Status 404 returned error can't find the container with id 544d4bff588275706104eab45cf608478f9c86e8706c1f47ed07547368daf0d6 Oct 30 06:45:07.213655 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.213632 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.214603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.214577 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9"] Oct 30 06:45:07.230669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.230642 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"openshift-service-ca.crt\"" Oct 30 06:45:07.231287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.231265 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c44730\"" Oct 30 06:45:07.231837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.231546 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"agc-cred-1868f6e16d0a\"" Oct 30 06:45:07.231837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.231627 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"payload-processing-external-model-e2e--9a02ae7b-dockercfg-sbxbp\"" Oct 30 06:45:07.231837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.231659 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"routing-overlay\"" Oct 30 06:45:07.231837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.231700 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"kube-root-ca.crt\"" Oct 30 06:45:07.231837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.231726 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt"] Oct 30 06:45:07.244980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.244950 5001 scope.go:122] "RemoveContainer" containerID="0ef4f74be3b4ef3e07c238b78499ce9dbc6fafeec2dcaa8a9d3dbabb13d676c3" Oct 30 06:45:07.274663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.274629 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.274868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.274722 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47l9v\" (UniqueName: \"kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-kube-api-access-47l9v\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.274868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.274816 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.275032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.275015 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.279352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.279277 5001 scope.go:122] "RemoveContainer" containerID="9bcce567bd8a81a49ed6de1a95cab4063c4a961b7eb82b8d497592cbfdc57638" Oct 30 06:45:07.279827 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.279789 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn_odh-ai-gateway-infra(fe6534e6-dec1-42b8-b780-83abc658be3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" Oct 30 06:45:07.282224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.282200 5001 scope.go:122] "RemoveContainer" containerID="54b6c2574201556010c88aa29a741c34619190052fc189a8e06fb5680411d749" Oct 30 06:45:07.282581 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.282555 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq_odh-ai-gateway-infra(1f71c0cf-a14f-4a6b-9067-9d60084a31c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" Oct 30 06:45:07.286215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.286196 5001 scope.go:122] "RemoveContainer" containerID="03a020a466e39b4ad78e16a44a6b842c51ce78d6a4931452c4bc81a3123e5dcf" Oct 30 06:45:07.286738 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.286716 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889045-st79h_odh-ai-gateway-infra(c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" Oct 30 06:45:07.288464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.288443 5001 scope.go:122] "RemoveContainer" containerID="987cc769b5fbcc3095e6e8300e4dd9e65f67bf10054fd13123946fc809fdfe55" Oct 30 06:45:07.288746 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.288722 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl_odh-ai-gateway-infra(557b0e7c-2d56-49fc-913b-a4da8206101a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" Oct 30 06:45:07.300574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.295415 5001 scope.go:122] "RemoveContainer" containerID="b98cacc880911c85bc719cf31ac0f6231098537162baa7d0e41830b4834cf425" Oct 30 06:45:07.300574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.296543 5001 scope.go:122] "RemoveContainer" containerID="84d4f51e86c442d2f1fe2dc7f448cbaf95a464740395d59d17049f670322c228" Oct 30 06:45:07.300574 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.296865 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5_odh-ai-gateway-infra(bb380de3-228b-448b-bc3b-848c1b638b11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" Oct 30 06:45:07.301741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.301720 5001 scope.go:122] "RemoveContainer" containerID="681d20e0c3b609dade73fab92c5d75896d87ea3a948f0c7ac76e6cc444201855" Oct 30 06:45:07.302319 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.302235 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8_odh-ai-gateway-infra(fc210264-c902-4f37-b47c-b25ce36a6482)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" Oct 30 06:45:07.303971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.303951 5001 scope.go:122] "RemoveContainer" containerID="3bb3da9cf0087186f2c2e86607d65514e3569b63d8750a2062e83c5424ae1c7d" Oct 30 06:45:07.304299 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.304275 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw_odh-ai-gateway-infra(70dfdbc8-cb44-4270-8108-11dddce181b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" Oct 30 06:45:07.308231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.308167 5001 scope.go:122] "RemoveContainer" containerID="d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb" Oct 30 06:45:07.308510 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.308485 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7_odh-ai-gateway-infra(12f95efa-c8f1-4e3a-a5db-cf529d3cad30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" Oct 30 06:45:07.310376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.310348 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" event={"ID":"a346576f-1151-4520-9dd7-dec9739f2032","Type":"ContainerStarted","Data":"544d4bff588275706104eab45cf608478f9c86e8706c1f47ed07547368daf0d6"} Oct 30 06:45:07.312104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.312085 5001 scope.go:122] "RemoveContainer" containerID="9c9841402efe937919b5d873b570c01fccc36b596685cc53ffe016ae4abc025d" Oct 30 06:45:07.312366 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.312345 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x_odh-ai-gateway-infra(30471359-76f9-4579-be9a-4dd7d3af77a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" Oct 30 06:45:07.315117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.315100 5001 scope.go:122] "RemoveContainer" containerID="5f08de7564f280c18f5d11d7c312289830bb7d24b71bedbec388df80e73c5672" Oct 30 06:45:07.315357 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.315339 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq_odh-ai-gateway-infra(50b32463-c8ed-417f-8718-c5ab4fd2f1b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" Oct 30 06:45:07.317499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.317481 5001 scope.go:122] "RemoveContainer" containerID="aa176122c3040e97cb533a1b9e961ee638bb0084050e670cbca43aed6564a027" Oct 30 06:45:07.317776 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.317753 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk_odh-ai-gateway-infra(72dc0962-b2a8-479e-986a-e36ebdfc5e75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" Oct 30 06:45:07.317912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.317895 5001 scope.go:122] "RemoveContainer" containerID="46a9a2d2436f13346046b1ded9780d3aa42d5b1b2ebd77b4a85b10c7fdbb1f0d" Oct 30 06:45:07.318203 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.318180 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt_odh-ai-gateway-infra(f74b178e-df08-4063-acdd-a0977b741211)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" podUID="f74b178e-df08-4063-acdd-a0977b741211" Oct 30 06:45:07.318342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.318328 5001 scope.go:122] "RemoveContainer" containerID="8a3e7897ec6ef7c1323bbe53e4b4318f76eea7cca854d21cdfc51932ceef87af" Oct 30 06:45:07.318469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.318449 5001 scope.go:122] "RemoveContainer" containerID="822b98090dcf0e4abd4459a34ac3ee68f3bdd8677399ec4d7996c880f462dd48" Oct 30 06:45:07.318607 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.318584 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw_odh-ai-gateway-infra(0d40aa92-6ad3-4340-807b-b4e4796baa3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" Oct 30 06:45:07.318779 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.318757 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf_odh-ai-gateway-infra(2ced181d-88b9-407f-8663-01b2f28072c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" Oct 30 06:45:07.326292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.326220 5001 scope.go:122] "RemoveContainer" containerID="1f7928f368743e81f7e8fdebc2aace13079590dbeaa9f1481abfbd0d14aa2361" Oct 30 06:45:07.382715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.376237 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.382715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.376407 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47l9v\" (UniqueName: \"kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-kube-api-access-47l9v\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.382715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.376542 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.382715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.377097 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.382715 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.377857 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c44730/agc-cred-1868f6e16d0a: secret "agc-cred-1868f6e16d0a" not found Oct 30 06:45:07.382715 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.377874 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9: secret "agc-cred-1868f6e16d0a" not found Oct 30 06:45:07.382715 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.377933 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-provider-credentials podName:1671ba31-314b-4cd3-9d31-6706ae443899 nodeName:}" failed. No retries permitted until 2026-10-30 06:45:07.877911973 +0000 UTC m=+3121.873868961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" (UID: "1671ba31-314b-4cd3-9d31-6706ae443899") : secret "agc-cred-1868f6e16d0a" not found Oct 30 06:45:07.382715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.378459 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.382715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.381278 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.457335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.451866 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47l9v\" (UniqueName: \"kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-kube-api-access-47l9v\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.611283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.611248 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt"] Oct 30 06:45:07.886480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:07.886370 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:07.886899 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.886875 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c44730/agc-cred-1868f6e16d0a: secret "agc-cred-1868f6e16d0a" not found Oct 30 06:45:07.886899 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.886898 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9: secret "agc-cred-1868f6e16d0a" not found Oct 30 06:45:07.887107 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:07.886966 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-provider-credentials podName:1671ba31-314b-4cd3-9d31-6706ae443899 nodeName:}" failed. No retries permitted until 2026-10-30 06:45:08.886945202 +0000 UTC m=+3122.882902207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" (UID: "1671ba31-314b-4cd3-9d31-6706ae443899") : secret "agc-cred-1868f6e16d0a" not found Oct 30 06:45:08.274910 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.274807 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:45:08.336510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:08.336473 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" event={"ID":"a346576f-1151-4520-9dd7-dec9739f2032","Type":"ContainerStarted","Data":"0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf"} Oct 30 06:45:08.336709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:08.336575 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:08.336767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:08.336738 5001 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" secret="" err="secret \"payload-processing-external-model-e2e--4e48ea35-dockercfg-v22pn\" not found" Oct 30 06:45:08.380952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:08.380896 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" podStartSLOduration=3.380883137 podStartE2EDuration="3.380883137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:08.378485881 +0000 UTC m=+3122.374442891" watchObservedRunningTime="2026-10-30 06:45:08.380883137 +0000 UTC m=+3122.376840146" Oct 30 06:45:08.409452 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.409419 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-plugins-e2e-extmod-a-88acce: configmap "payload-processing-external-model-plugins-e2e-extmod-a-88acce" not found Oct 30 06:45:08.409452 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.409446 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-39512e62d644: secret "agc-cred-39512e62d644" not found Oct 30 06:45:08.409715 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.409471 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt: secret "agc-cred-39512e62d644" not found Oct 30 06:45:08.409715 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.409518 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume podName:a346576f-1151-4520-9dd7-dec9739f2032 nodeName:}" failed. No retries permitted until 2026-10-30 06:45:08.909498674 +0000 UTC m=+3122.905455677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" (UID: "a346576f-1151-4520-9dd7-dec9739f2032") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-88acce" not found Oct 30 06:45:08.409715 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.409539 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials podName:a346576f-1151-4520-9dd7-dec9739f2032 nodeName:}" failed. No retries permitted until 2026-10-30 06:45:08.909530118 +0000 UTC m=+3122.905487110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" (UID: "a346576f-1151-4520-9dd7-dec9739f2032") : secret "agc-cred-39512e62d644" not found Oct 30 06:45:08.588430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:08.588340 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9"] Oct 30 06:45:08.589245 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.589204 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" podUID="1671ba31-314b-4cd3-9d31-6706ae443899" Oct 30 06:45:08.923965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:08.923926 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:08.924138 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.924107 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c44730/agc-cred-1868f6e16d0a: secret "agc-cred-1868f6e16d0a" not found Oct 30 06:45:08.924138 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.924135 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9: secret "agc-cred-1868f6e16d0a" not found Oct 30 06:45:08.924257 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.924136 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-plugins-e2e-extmod-a-88acce: configmap "payload-processing-external-model-plugins-e2e-extmod-a-88acce" not found Oct 30 06:45:08.924257 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.924192 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume podName:a346576f-1151-4520-9dd7-dec9739f2032 nodeName:}" failed. No retries permitted until 2026-10-30 06:45:09.924177327 +0000 UTC m=+3123.920134319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" (UID: "a346576f-1151-4520-9dd7-dec9739f2032") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-88acce" not found Oct 30 06:45:08.924257 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.924212 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-provider-credentials podName:1671ba31-314b-4cd3-9d31-6706ae443899 nodeName:}" failed. No retries permitted until 2026-10-30 06:45:10.924205669 +0000 UTC m=+3124.920162660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" (UID: "1671ba31-314b-4cd3-9d31-6706ae443899") : secret "agc-cred-1868f6e16d0a" not found Oct 30 06:45:08.924257 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.924236 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-39512e62d644: secret "agc-cred-39512e62d644" not found Oct 30 06:45:08.924257 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.924256 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt: secret "agc-cred-39512e62d644" not found Oct 30 06:45:08.924504 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:08.924330 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials podName:a346576f-1151-4520-9dd7-dec9739f2032 nodeName:}" failed. No retries permitted until 2026-10-30 06:45:09.9243006 +0000 UTC m=+3123.920257591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" (UID: "a346576f-1151-4520-9dd7-dec9739f2032") : secret "agc-cred-39512e62d644" not found Oct 30 06:45:09.348010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:09.347912 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:09.360415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:09.360385 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:09.447982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:09.447948 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-kube-api-access-47l9v\" (UniqueName: \"kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-kube-api-access-47l9v\") pod \"1671ba31-314b-4cd3-9d31-6706ae443899\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " Oct 30 06:45:09.448340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:09.448300 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-plugins-config-volume\") pod \"1671ba31-314b-4cd3-9d31-6706ae443899\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " Oct 30 06:45:09.448413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:09.448358 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-routing-overlay-volume\") pod \"1671ba31-314b-4cd3-9d31-6706ae443899\" (UID: \"1671ba31-314b-4cd3-9d31-6706ae443899\") " Oct 30 06:45:09.448727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:09.448694 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-plugins-config-volume" pod "1671ba31-314b-4cd3-9d31-6706ae443899" (UID: "1671ba31-314b-4cd3-9d31-6706ae443899"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:09.448876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:09.448702 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-routing-overlay-volume" pod "1671ba31-314b-4cd3-9d31-6706ae443899" (UID: "1671ba31-314b-4cd3-9d31-6706ae443899"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:09.449147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:09.449127 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:09.449209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:09.449150 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1671ba31-314b-4cd3-9d31-6706ae443899-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:09.450577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:09.450547 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-kube-api-access-47l9v" pod "1671ba31-314b-4cd3-9d31-6706ae443899" (UID: "1671ba31-314b-4cd3-9d31-6706ae443899"). InnerVolumeSpecName "kube-api-access-47l9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:09.551888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:09.551781 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47l9v\" (UniqueName: \"kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-kube-api-access-47l9v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:09.958925 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:09.958888 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-plugins-e2e-extmod-a-88acce: configmap "payload-processing-external-model-plugins-e2e-extmod-a-88acce" not found Oct 30 06:45:09.959133 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:09.958963 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-39512e62d644: secret "agc-cred-39512e62d644" not found Oct 30 06:45:09.959133 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:09.958988 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt: secret "agc-cred-39512e62d644" not found Oct 30 06:45:09.959133 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:09.958967 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume podName:a346576f-1151-4520-9dd7-dec9739f2032 nodeName:}" failed. No retries permitted until 2026-10-30 06:45:11.958951554 +0000 UTC m=+3125.954908542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" (UID: "a346576f-1151-4520-9dd7-dec9739f2032") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-88acce" not found Oct 30 06:45:09.959133 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:09.959045 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials podName:a346576f-1151-4520-9dd7-dec9739f2032 nodeName:}" failed. No retries permitted until 2026-10-30 06:45:11.959028262 +0000 UTC m=+3125.954985255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" (UID: "a346576f-1151-4520-9dd7-dec9739f2032") : secret "agc-cred-39512e62d644" not found Oct 30 06:45:10.361700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:10.361663 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9" Oct 30 06:45:10.446889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:10.446852 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9"] Oct 30 06:45:10.452928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:10.452882 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c992dzh9"] Oct 30 06:45:10.571054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:10.571014 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1671ba31-314b-4cd3-9d31-6706ae443899-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:11.121026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:11.120989 5001 scope.go:122] "RemoveContainer" containerID="0996e849b4a17c2aec0bd625d31a4010dcce6e910edf81475a0b51b3a9358daa" Oct 30 06:45:11.133649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:11.133626 5001 scope.go:122] "RemoveContainer" containerID="a22f1ee71ed2fabf13080142e77960dd15bc44a18104afaf70dd8474165264a4" Oct 30 06:45:11.145956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:11.145934 5001 scope.go:122] "RemoveContainer" containerID="8eef61748104fb72621f9ad0022fa43d3817730c409bc406e22c46b70e393e69" Oct 30 06:45:12.005186 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:12.005152 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-plugins-e2e-extmod-a-88acce: configmap "payload-processing-external-model-plugins-e2e-extmod-a-88acce" not found Oct 30 06:45:12.005637 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:12.005241 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume podName:a346576f-1151-4520-9dd7-dec9739f2032 nodeName:}" failed. No retries permitted until 2026-10-30 06:45:16.005220779 +0000 UTC m=+3130.001177768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" (UID: "a346576f-1151-4520-9dd7-dec9739f2032") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-88acce" not found Oct 30 06:45:12.005637 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:12.005254 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-39512e62d644: secret "agc-cred-39512e62d644" not found Oct 30 06:45:12.005637 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:12.005270 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt: secret "agc-cred-39512e62d644" not found Oct 30 06:45:12.005637 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:12.005350 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials podName:a346576f-1151-4520-9dd7-dec9739f2032 nodeName:}" failed. No retries permitted until 2026-10-30 06:45:16.005295908 +0000 UTC m=+3130.001252910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" (UID: "a346576f-1151-4520-9dd7-dec9739f2032") : secret "agc-cred-39512e62d644" not found Oct 30 06:45:12.120856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:12.120822 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1671ba31-314b-4cd3-9d31-6706ae443899" path="/var/lib/kubelet/pods/1671ba31-314b-4cd3-9d31-6706ae443899/volumes" Oct 30 06:45:14.348129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.348086 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" podUID="a346576f-1151-4520-9dd7-dec9739f2032" containerName="payload-processing" containerID="cri-o://0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf" gracePeriod=25 Oct 30 06:45:14.737011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.736984 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:14.767098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.767066 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-routing-overlay-volume\") pod \"a346576f-1151-4520-9dd7-dec9739f2032\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " Oct 30 06:45:14.767288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.767269 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-kube-api-access-9gjmj\" (UniqueName: \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-kube-api-access-9gjmj\") pod \"a346576f-1151-4520-9dd7-dec9739f2032\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " Oct 30 06:45:14.767499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.767399 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume\") pod \"a346576f-1151-4520-9dd7-dec9739f2032\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " Oct 30 06:45:14.767599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.767494 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-routing-overlay-volume" pod "a346576f-1151-4520-9dd7-dec9739f2032" (UID: "a346576f-1151-4520-9dd7-dec9739f2032"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:14.767599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.767510 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials\") pod \"a346576f-1151-4520-9dd7-dec9739f2032\" (UID: \"a346576f-1151-4520-9dd7-dec9739f2032\") " Oct 30 06:45:14.768061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.768008 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume" pod "a346576f-1151-4520-9dd7-dec9739f2032" (UID: "a346576f-1151-4520-9dd7-dec9739f2032"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:14.768290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.768269 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:14.768404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.768296 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a346576f-1151-4520-9dd7-dec9739f2032-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:14.770021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.769991 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials" pod "a346576f-1151-4520-9dd7-dec9739f2032" (UID: "a346576f-1151-4520-9dd7-dec9739f2032"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:14.770158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.770133 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-kube-api-access-9gjmj" pod "a346576f-1151-4520-9dd7-dec9739f2032" (UID: "a346576f-1151-4520-9dd7-dec9739f2032"). InnerVolumeSpecName "kube-api-access-9gjmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:14.870595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.870560 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gjmj\" (UniqueName: \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-kube-api-access-9gjmj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:14.870595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:14.870590 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a346576f-1151-4520-9dd7-dec9739f2032-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:15.469968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:15.469920 5001 generic.go:356] "Generic (PLEG): container finished" podID="a346576f-1151-4520-9dd7-dec9739f2032" containerID="0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf" exitCode=0 Oct 30 06:45:15.469968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:15.469968 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" Oct 30 06:45:15.470623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:15.470002 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" event={"ID":"a346576f-1151-4520-9dd7-dec9739f2032","Type":"ContainerDied","Data":"0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf"} Oct 30 06:45:15.470623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:15.470080 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt" event={"ID":"a346576f-1151-4520-9dd7-dec9739f2032","Type":"ContainerDied","Data":"544d4bff588275706104eab45cf608478f9c86e8706c1f47ed07547368daf0d6"} Oct 30 06:45:15.470623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:15.470111 5001 scope.go:122] "RemoveContainer" containerID="0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf" Oct 30 06:45:15.484910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:15.484886 5001 scope.go:122] "RemoveContainer" containerID="0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf" Oct 30 06:45:15.485240 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:15.485217 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf\": container with ID starting with 0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf not found: ID does not exist" containerID="0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf" Oct 30 06:45:15.486556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:15.485252 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf"} err="failed to get container status \"0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf\": rpc error: code = NotFound desc = could not find container \"0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf\": container with ID starting with 0f24ed4e24ad6125c3b8de0e73af9b86e2d6103f8ce23b34a3eb7b96a727b7cf not found: ID does not exist" Oct 30 06:45:15.528555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:15.528525 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt"] Oct 30 06:45:15.542440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:15.542411 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6ff6x4xbt"] Oct 30 06:45:16.121632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:16.121597 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a346576f-1151-4520-9dd7-dec9739f2032" path="/var/lib/kubelet/pods/a346576f-1151-4520-9dd7-dec9739f2032/volumes" Oct 30 06:45:16.122481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:16.122465 5001 scope.go:122] "RemoveContainer" containerID="54b6c2574201556010c88aa29a741c34619190052fc189a8e06fb5680411d749" Oct 30 06:45:17.110253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:17.110228 5001 scope.go:122] "RemoveContainer" containerID="9c9841402efe937919b5d873b570c01fccc36b596685cc53ffe016ae4abc025d" Oct 30 06:45:17.110656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:17.110454 5001 scope.go:122] "RemoveContainer" containerID="8a3e7897ec6ef7c1323bbe53e4b4318f76eea7cca854d21cdfc51932ceef87af" Oct 30 06:45:17.110656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:17.110590 5001 scope.go:122] "RemoveContainer" containerID="dd44523be1aeb68e1901b4d304511366b8c70aac10bc74aae9b338c63b344f40" Oct 30 06:45:17.545872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:17.545840 5001 generic.go:356] "Generic (PLEG): container finished" podID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerID="6aaf90ec22bbfbb968b353519756955e0a8c7389787e7f26621fd8ffec260a4d" exitCode=60 Oct 30 06:45:17.546069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:17.545913 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" event={"ID":"1f71c0cf-a14f-4a6b-9067-9d60084a31c6","Type":"ContainerDied","Data":"6aaf90ec22bbfbb968b353519756955e0a8c7389787e7f26621fd8ffec260a4d"} Oct 30 06:45:17.546069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:17.545970 5001 scope.go:122] "RemoveContainer" containerID="54b6c2574201556010c88aa29a741c34619190052fc189a8e06fb5680411d749" Oct 30 06:45:17.546464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:17.546443 5001 scope.go:122] "RemoveContainer" containerID="6aaf90ec22bbfbb968b353519756955e0a8c7389787e7f26621fd8ffec260a4d" Oct 30 06:45:17.547000 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:17.546887 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq_odh-ai-gateway-infra(1f71c0cf-a14f-4a6b-9067-9d60084a31c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" Oct 30 06:45:18.121177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.121148 5001 scope.go:122] "RemoveContainer" containerID="822b98090dcf0e4abd4459a34ac3ee68f3bdd8677399ec4d7996c880f462dd48" Oct 30 06:45:18.121640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.121323 5001 scope.go:122] "RemoveContainer" containerID="03a020a466e39b4ad78e16a44a6b842c51ce78d6a4931452c4bc81a3123e5dcf" Oct 30 06:45:18.121640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.121484 5001 scope.go:122] "RemoveContainer" containerID="5f08de7564f280c18f5d11d7c312289830bb7d24b71bedbec388df80e73c5672" Oct 30 06:45:18.561048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.561013 5001 generic.go:356] "Generic (PLEG): container finished" podID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerID="7290d16e1fda72aef47ef461c0187f3cc36a0776dca2cbfd9678232e8a451222" exitCode=60 Oct 30 06:45:18.561263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.561186 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" event={"ID":"30471359-76f9-4579-be9a-4dd7d3af77a0","Type":"ContainerDied","Data":"7290d16e1fda72aef47ef461c0187f3cc36a0776dca2cbfd9678232e8a451222"} Oct 30 06:45:18.561263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.561232 5001 scope.go:122] "RemoveContainer" containerID="9c9841402efe937919b5d873b570c01fccc36b596685cc53ffe016ae4abc025d" Oct 30 06:45:18.561690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.561672 5001 scope.go:122] "RemoveContainer" containerID="7290d16e1fda72aef47ef461c0187f3cc36a0776dca2cbfd9678232e8a451222" Oct 30 06:45:18.562081 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:18.562060 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x_odh-ai-gateway-infra(30471359-76f9-4579-be9a-4dd7d3af77a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" Oct 30 06:45:18.569143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.569111 5001 generic.go:356] "Generic (PLEG): container finished" podID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerID="fc685ed256c61158b1c442ce1950e67b3e08e342dec6bf00e47c140c4c85c484" exitCode=60 Oct 30 06:45:18.569352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.569227 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" event={"ID":"0d40aa92-6ad3-4340-807b-b4e4796baa3d","Type":"ContainerDied","Data":"fc685ed256c61158b1c442ce1950e67b3e08e342dec6bf00e47c140c4c85c484"} Oct 30 06:45:18.569811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.569788 5001 scope.go:122] "RemoveContainer" containerID="fc685ed256c61158b1c442ce1950e67b3e08e342dec6bf00e47c140c4c85c484" Oct 30 06:45:18.570274 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:18.570251 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw_odh-ai-gateway-infra(0d40aa92-6ad3-4340-807b-b4e4796baa3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" Oct 30 06:45:18.572612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.572590 5001 generic.go:356] "Generic (PLEG): container finished" podID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerID="9568a64b4b715f2ce41b503bca31c4594eef3ccb46d4408cabd1b1e67ee6f397" exitCode=60 Oct 30 06:45:18.572737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.572669 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" event={"ID":"62f0f671-aaa1-4a3f-a267-19d668b62f80","Type":"ContainerDied","Data":"9568a64b4b715f2ce41b503bca31c4594eef3ccb46d4408cabd1b1e67ee6f397"} Oct 30 06:45:18.573158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.573134 5001 scope.go:122] "RemoveContainer" containerID="9568a64b4b715f2ce41b503bca31c4594eef3ccb46d4408cabd1b1e67ee6f397" Oct 30 06:45:18.573586 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:18.573562 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs_odh-ai-gateway-infra(62f0f671-aaa1-4a3f-a267-19d668b62f80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" Oct 30 06:45:18.601877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.601848 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq"] Oct 30 06:45:18.615862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.615741 5001 scope.go:122] "RemoveContainer" containerID="8a3e7897ec6ef7c1323bbe53e4b4318f76eea7cca854d21cdfc51932ceef87af" Oct 30 06:45:18.639672 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.639639 5001 scope.go:122] "RemoveContainer" containerID="dd44523be1aeb68e1901b4d304511366b8c70aac10bc74aae9b338c63b344f40" Oct 30 06:45:18.922168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.922139 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" Oct 30 06:45:18.983111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.983075 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-maas-api-service-ca\") pod \"1f71c0cf-a14f-4a6b-9067-9d60084a31c6\" (UID: \"1f71c0cf-a14f-4a6b-9067-9d60084a31c6\") " Oct 30 06:45:18.983508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.983481 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-kube-api-access-5bmc5\" (UniqueName: \"kubernetes.io/projected/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-kube-api-access-5bmc5\") pod \"1f71c0cf-a14f-4a6b-9067-9d60084a31c6\" (UID: \"1f71c0cf-a14f-4a6b-9067-9d60084a31c6\") " Oct 30 06:45:18.983635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.983557 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-maas-api-service-ca" pod "1f71c0cf-a14f-4a6b-9067-9d60084a31c6" (UID: "1f71c0cf-a14f-4a6b-9067-9d60084a31c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:18.989757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.984149 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:18.989757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:18.986442 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-kube-api-access-5bmc5" pod "1f71c0cf-a14f-4a6b-9067-9d60084a31c6" (UID: "1f71c0cf-a14f-4a6b-9067-9d60084a31c6"). InnerVolumeSpecName "kube-api-access-5bmc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:19.086231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.086139 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bmc5\" (UniqueName: \"kubernetes.io/projected/1f71c0cf-a14f-4a6b-9067-9d60084a31c6-kube-api-access-5bmc5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:19.110192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.110163 5001 scope.go:122] "RemoveContainer" containerID="987cc769b5fbcc3095e6e8300e4dd9e65f67bf10054fd13123946fc809fdfe55" Oct 30 06:45:19.110414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.110379 5001 scope.go:122] "RemoveContainer" containerID="46a9a2d2436f13346046b1ded9780d3aa42d5b1b2ebd77b4a85b10c7fdbb1f0d" Oct 30 06:45:19.110549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.110525 5001 scope.go:122] "RemoveContainer" containerID="681d20e0c3b609dade73fab92c5d75896d87ea3a948f0c7ac76e6cc444201855" Oct 30 06:45:19.110626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.110604 5001 scope.go:122] "RemoveContainer" containerID="3bb3da9cf0087186f2c2e86607d65514e3569b63d8750a2062e83c5424ae1c7d" Oct 30 06:45:19.110748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.110727 5001 scope.go:122] "RemoveContainer" containerID="aa176122c3040e97cb533a1b9e961ee638bb0084050e670cbca43aed6564a027" Oct 30 06:45:19.593252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.593226 5001 generic.go:356] "Generic (PLEG): container finished" podID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerID="7d1872f513de9d683ee960557b24b5976da095b0a69b2cba04a3d22bb734528d" exitCode=60 Oct 30 06:45:19.593737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.593485 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" event={"ID":"50b32463-c8ed-417f-8718-c5ab4fd2f1b7","Type":"ContainerDied","Data":"7d1872f513de9d683ee960557b24b5976da095b0a69b2cba04a3d22bb734528d"} Oct 30 06:45:19.593737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.593557 5001 scope.go:122] "RemoveContainer" containerID="5f08de7564f280c18f5d11d7c312289830bb7d24b71bedbec388df80e73c5672" Oct 30 06:45:19.594427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.594113 5001 scope.go:122] "RemoveContainer" containerID="7d1872f513de9d683ee960557b24b5976da095b0a69b2cba04a3d22bb734528d" Oct 30 06:45:19.594793 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:19.594762 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq_odh-ai-gateway-infra(50b32463-c8ed-417f-8718-c5ab4fd2f1b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" Oct 30 06:45:19.597169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.596575 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" Oct 30 06:45:19.597169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.596594 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq" event={"ID":"1f71c0cf-a14f-4a6b-9067-9d60084a31c6","Type":"ContainerDied","Data":"63663646c54c1d17562d8b08c9738889d773bda2ecc13f58015c57540569bca2"} Oct 30 06:45:19.599859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.599004 5001 generic.go:356] "Generic (PLEG): container finished" podID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerID="48f2ff4bc6e58547543f3efc6dc75d60a20dfeb13735bfe15554ccc28e318dd5" exitCode=60 Oct 30 06:45:19.599859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.599080 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" event={"ID":"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08","Type":"ContainerDied","Data":"48f2ff4bc6e58547543f3efc6dc75d60a20dfeb13735bfe15554ccc28e318dd5"} Oct 30 06:45:19.599859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.599618 5001 scope.go:122] "RemoveContainer" containerID="48f2ff4bc6e58547543f3efc6dc75d60a20dfeb13735bfe15554ccc28e318dd5" Oct 30 06:45:19.600382 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:19.600354 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889045-st79h_odh-ai-gateway-infra(c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" Oct 30 06:45:19.601967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.601940 5001 generic.go:356] "Generic (PLEG): container finished" podID="2ced181d-88b9-407f-8663-01b2f28072c2" containerID="6b3701c1944c527f08908463c8a89154006377c3f3af5bd24b3c5d72165349dc" exitCode=60 Oct 30 06:45:19.602113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.602095 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" event={"ID":"2ced181d-88b9-407f-8663-01b2f28072c2","Type":"ContainerDied","Data":"6b3701c1944c527f08908463c8a89154006377c3f3af5bd24b3c5d72165349dc"} Oct 30 06:45:19.602559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.602542 5001 scope.go:122] "RemoveContainer" containerID="6b3701c1944c527f08908463c8a89154006377c3f3af5bd24b3c5d72165349dc" Oct 30 06:45:19.602940 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:19.602809 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf_odh-ai-gateway-infra(2ced181d-88b9-407f-8663-01b2f28072c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" Oct 30 06:45:19.618358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.618260 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x"] Oct 30 06:45:19.681952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.681833 5001 scope.go:122] "RemoveContainer" containerID="6aaf90ec22bbfbb968b353519756955e0a8c7389787e7f26621fd8ffec260a4d" Oct 30 06:45:19.694685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.694649 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw"] Oct 30 06:45:19.714938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.714907 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs"] Oct 30 06:45:19.724160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.724132 5001 scope.go:122] "RemoveContainer" containerID="03a020a466e39b4ad78e16a44a6b842c51ce78d6a4931452c4bc81a3123e5dcf" Oct 30 06:45:19.817518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.817491 5001 scope.go:122] "RemoveContainer" containerID="822b98090dcf0e4abd4459a34ac3ee68f3bdd8677399ec4d7996c880f462dd48" Oct 30 06:45:19.927960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.927936 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" Oct 30 06:45:19.989216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:19.989184 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq"] Oct 30 06:45:20.001813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.001766 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-868b1f-29889045-xpzlq"] Oct 30 06:45:20.015072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.015040 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30471359-76f9-4579-be9a-4dd7d3af77a0-kube-api-access-2hbht\" (UniqueName: \"kubernetes.io/projected/30471359-76f9-4579-be9a-4dd7d3af77a0-kube-api-access-2hbht\") pod \"30471359-76f9-4579-be9a-4dd7d3af77a0\" (UID: \"30471359-76f9-4579-be9a-4dd7d3af77a0\") " Oct 30 06:45:20.015254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.015173 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30471359-76f9-4579-be9a-4dd7d3af77a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30471359-76f9-4579-be9a-4dd7d3af77a0-maas-api-service-ca\") pod \"30471359-76f9-4579-be9a-4dd7d3af77a0\" (UID: \"30471359-76f9-4579-be9a-4dd7d3af77a0\") " Oct 30 06:45:20.016053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.016021 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30471359-76f9-4579-be9a-4dd7d3af77a0-maas-api-service-ca" pod "30471359-76f9-4579-be9a-4dd7d3af77a0" (UID: "30471359-76f9-4579-be9a-4dd7d3af77a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:20.019630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.019605 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30471359-76f9-4579-be9a-4dd7d3af77a0-kube-api-access-2hbht" pod "30471359-76f9-4579-be9a-4dd7d3af77a0" (UID: "30471359-76f9-4579-be9a-4dd7d3af77a0"). InnerVolumeSpecName "kube-api-access-2hbht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:20.086752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.086718 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" Oct 30 06:45:20.111529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.110245 5001 scope.go:122] "RemoveContainer" containerID="9bcce567bd8a81a49ed6de1a95cab4063c4a961b7eb82b8d497592cbfdc57638" Oct 30 06:45:20.111729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.111633 5001 scope.go:122] "RemoveContainer" containerID="84d4f51e86c442d2f1fe2dc7f448cbaf95a464740395d59d17049f670322c228" Oct 30 06:45:20.116777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.116748 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f0f671-aaa1-4a3f-a267-19d668b62f80-kube-api-access-xjftg\" (UniqueName: \"kubernetes.io/projected/62f0f671-aaa1-4a3f-a267-19d668b62f80-kube-api-access-xjftg\") pod \"62f0f671-aaa1-4a3f-a267-19d668b62f80\" (UID: \"62f0f671-aaa1-4a3f-a267-19d668b62f80\") " Oct 30 06:45:20.116921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.116827 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f0f671-aaa1-4a3f-a267-19d668b62f80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f0f671-aaa1-4a3f-a267-19d668b62f80-maas-api-service-ca\") pod \"62f0f671-aaa1-4a3f-a267-19d668b62f80\" (UID: \"62f0f671-aaa1-4a3f-a267-19d668b62f80\") " Oct 30 06:45:20.117617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.117582 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hbht\" (UniqueName: \"kubernetes.io/projected/30471359-76f9-4579-be9a-4dd7d3af77a0-kube-api-access-2hbht\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.117617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.117610 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30471359-76f9-4579-be9a-4dd7d3af77a0-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.119103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.119078 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f0f671-aaa1-4a3f-a267-19d668b62f80-maas-api-service-ca" pod "62f0f671-aaa1-4a3f-a267-19d668b62f80" (UID: "62f0f671-aaa1-4a3f-a267-19d668b62f80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:20.122518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.122492 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f0f671-aaa1-4a3f-a267-19d668b62f80-kube-api-access-xjftg" pod "62f0f671-aaa1-4a3f-a267-19d668b62f80" (UID: "62f0f671-aaa1-4a3f-a267-19d668b62f80"). InnerVolumeSpecName "kube-api-access-xjftg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:20.127160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.127140 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" path="/var/lib/kubelet/pods/1f71c0cf-a14f-4a6b-9067-9d60084a31c6/volumes" Oct 30 06:45:20.220692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.220659 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjftg\" (UniqueName: \"kubernetes.io/projected/62f0f671-aaa1-4a3f-a267-19d668b62f80-kube-api-access-xjftg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.221029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.221007 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f0f671-aaa1-4a3f-a267-19d668b62f80-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.312445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.312421 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" Oct 30 06:45:20.322192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.322166 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d40aa92-6ad3-4340-807b-b4e4796baa3d-kube-api-access-hpc84\" (UniqueName: \"kubernetes.io/projected/0d40aa92-6ad3-4340-807b-b4e4796baa3d-kube-api-access-hpc84\") pod \"0d40aa92-6ad3-4340-807b-b4e4796baa3d\" (UID: \"0d40aa92-6ad3-4340-807b-b4e4796baa3d\") " Oct 30 06:45:20.322399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.322385 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d40aa92-6ad3-4340-807b-b4e4796baa3d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d40aa92-6ad3-4340-807b-b4e4796baa3d-maas-api-service-ca\") pod \"0d40aa92-6ad3-4340-807b-b4e4796baa3d\" (UID: \"0d40aa92-6ad3-4340-807b-b4e4796baa3d\") " Oct 30 06:45:20.322730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.322709 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d40aa92-6ad3-4340-807b-b4e4796baa3d-maas-api-service-ca" pod "0d40aa92-6ad3-4340-807b-b4e4796baa3d" (UID: "0d40aa92-6ad3-4340-807b-b4e4796baa3d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:20.322964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.322947 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d40aa92-6ad3-4340-807b-b4e4796baa3d-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.360148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.359520 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d40aa92-6ad3-4340-807b-b4e4796baa3d-kube-api-access-hpc84" pod "0d40aa92-6ad3-4340-807b-b4e4796baa3d" (UID: "0d40aa92-6ad3-4340-807b-b4e4796baa3d"). InnerVolumeSpecName "kube-api-access-hpc84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:20.424997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.424957 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpc84\" (UniqueName: \"kubernetes.io/projected/0d40aa92-6ad3-4340-807b-b4e4796baa3d-kube-api-access-hpc84\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.620420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.620383 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" event={"ID":"30471359-76f9-4579-be9a-4dd7d3af77a0","Type":"ContainerDied","Data":"52d2df6c3a591ca231df3ba0d50d65b8246e1eddb43bac42612e97a5a9370f32"} Oct 30 06:45:20.620974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.620461 5001 scope.go:122] "RemoveContainer" containerID="7290d16e1fda72aef47ef461c0187f3cc36a0776dca2cbfd9678232e8a451222" Oct 30 06:45:20.621016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.620979 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x" Oct 30 06:45:20.635318 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.635218 5001 generic.go:356] "Generic (PLEG): container finished" podID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerID="a169eef7a7008a01441b879a9747932320c7c2ab818ac27f5071fd7080ea1528" exitCode=60 Oct 30 06:45:20.635489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.635443 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" event={"ID":"72dc0962-b2a8-479e-986a-e36ebdfc5e75","Type":"ContainerDied","Data":"a169eef7a7008a01441b879a9747932320c7c2ab818ac27f5071fd7080ea1528"} Oct 30 06:45:20.635955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.635934 5001 scope.go:122] "RemoveContainer" containerID="a169eef7a7008a01441b879a9747932320c7c2ab818ac27f5071fd7080ea1528" Oct 30 06:45:20.636457 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:20.636431 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk_odh-ai-gateway-infra(72dc0962-b2a8-479e-986a-e36ebdfc5e75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" Oct 30 06:45:20.642620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.642598 5001 generic.go:356] "Generic (PLEG): container finished" podID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerID="0fa56e3acfdec1791e4ea18c5b70233be6d4eb9ace9e1e5e2a88ae113644516c" exitCode=60 Oct 30 06:45:20.642721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.642669 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" event={"ID":"557b0e7c-2d56-49fc-913b-a4da8206101a","Type":"ContainerDied","Data":"0fa56e3acfdec1791e4ea18c5b70233be6d4eb9ace9e1e5e2a88ae113644516c"} Oct 30 06:45:20.643131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.643109 5001 scope.go:122] "RemoveContainer" containerID="0fa56e3acfdec1791e4ea18c5b70233be6d4eb9ace9e1e5e2a88ae113644516c" Oct 30 06:45:20.643490 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:20.643467 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl_odh-ai-gateway-infra(557b0e7c-2d56-49fc-913b-a4da8206101a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" Oct 30 06:45:20.646532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.646238 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" event={"ID":"0d40aa92-6ad3-4340-807b-b4e4796baa3d","Type":"ContainerDied","Data":"80b6786380f0717b035293dce6212a264f87bb667546a4027e4b70fa14c6d1c6"} Oct 30 06:45:20.646532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.646372 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw" Oct 30 06:45:20.656663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.656629 5001 generic.go:356] "Generic (PLEG): container finished" podID="f74b178e-df08-4063-acdd-a0977b741211" containerID="4ed62793a8545d232f7b743335f8f42ef5f4f513fbe5883eae805546aa1ea9c7" exitCode=60 Oct 30 06:45:20.656807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.656727 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" event={"ID":"f74b178e-df08-4063-acdd-a0977b741211","Type":"ContainerDied","Data":"4ed62793a8545d232f7b743335f8f42ef5f4f513fbe5883eae805546aa1ea9c7"} Oct 30 06:45:20.657173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.657155 5001 scope.go:122] "RemoveContainer" containerID="4ed62793a8545d232f7b743335f8f42ef5f4f513fbe5883eae805546aa1ea9c7" Oct 30 06:45:20.659698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.659671 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq"] Oct 30 06:45:20.660239 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:20.660216 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt_odh-ai-gateway-infra(f74b178e-df08-4063-acdd-a0977b741211)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" podUID="f74b178e-df08-4063-acdd-a0977b741211" Oct 30 06:45:20.662177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.662150 5001 generic.go:356] "Generic (PLEG): container finished" podID="fc210264-c902-4f37-b47c-b25ce36a6482" containerID="91797fb66c05bffdb33d1883788c2d6043e6239d8c9d5df006a9b09c5695dcbc" exitCode=60 Oct 30 06:45:20.662297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.662226 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" event={"ID":"fc210264-c902-4f37-b47c-b25ce36a6482","Type":"ContainerDied","Data":"91797fb66c05bffdb33d1883788c2d6043e6239d8c9d5df006a9b09c5695dcbc"} Oct 30 06:45:20.662887 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.662743 5001 scope.go:122] "RemoveContainer" containerID="91797fb66c05bffdb33d1883788c2d6043e6239d8c9d5df006a9b09c5695dcbc" Oct 30 06:45:20.663149 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:20.663099 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8_odh-ai-gateway-infra(fc210264-c902-4f37-b47c-b25ce36a6482)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" Oct 30 06:45:20.673722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.673692 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" Oct 30 06:45:20.674407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.674380 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs" event={"ID":"62f0f671-aaa1-4a3f-a267-19d668b62f80","Type":"ContainerDied","Data":"4369daec22e3d669ad298cf8fc85841d6697ab571975c38a2c49b73da5d22ae1"} Oct 30 06:45:20.677176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.677148 5001 generic.go:356] "Generic (PLEG): container finished" podID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerID="0192b87a70135fada2c15aef621aca863a9f3f7ff7d18ec5c718e3520c290925" exitCode=60 Oct 30 06:45:20.677299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.677231 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" event={"ID":"70dfdbc8-cb44-4270-8108-11dddce181b9","Type":"ContainerDied","Data":"0192b87a70135fada2c15aef621aca863a9f3f7ff7d18ec5c718e3520c290925"} Oct 30 06:45:20.677774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.677754 5001 scope.go:122] "RemoveContainer" containerID="0192b87a70135fada2c15aef621aca863a9f3f7ff7d18ec5c718e3520c290925" Oct 30 06:45:20.678133 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:20.678085 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw_odh-ai-gateway-infra(70dfdbc8-cb44-4270-8108-11dddce181b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" Oct 30 06:45:20.731993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.731951 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h"] Oct 30 06:45:20.824383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.824350 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf"] Oct 30 06:45:20.889952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.889921 5001 scope.go:122] "RemoveContainer" containerID="aa176122c3040e97cb533a1b9e961ee638bb0084050e670cbca43aed6564a027" Oct 30 06:45:20.890286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.890204 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" Oct 30 06:45:20.928268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.927967 5001 scope.go:122] "RemoveContainer" containerID="987cc769b5fbcc3095e6e8300e4dd9e65f67bf10054fd13123946fc809fdfe55" Oct 30 06:45:20.947125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.947012 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-kube-api-access-2flf6\" (UniqueName: \"kubernetes.io/projected/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-kube-api-access-2flf6\") pod \"50b32463-c8ed-417f-8718-c5ab4fd2f1b7\" (UID: \"50b32463-c8ed-417f-8718-c5ab4fd2f1b7\") " Oct 30 06:45:20.947125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.947085 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-maas-api-service-ca\") pod \"50b32463-c8ed-417f-8718-c5ab4fd2f1b7\" (UID: \"50b32463-c8ed-417f-8718-c5ab4fd2f1b7\") " Oct 30 06:45:20.947929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.947880 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-maas-api-service-ca" pod "50b32463-c8ed-417f-8718-c5ab4fd2f1b7" (UID: "50b32463-c8ed-417f-8718-c5ab4fd2f1b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:20.948316 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.948279 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.953661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.952586 5001 scope.go:122] "RemoveContainer" containerID="fc685ed256c61158b1c442ce1950e67b3e08e342dec6bf00e47c140c4c85c484" Oct 30 06:45:20.970541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.961997 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-kube-api-access-2flf6" pod "50b32463-c8ed-417f-8718-c5ab4fd2f1b7" (UID: "50b32463-c8ed-417f-8718-c5ab4fd2f1b7"). InnerVolumeSpecName "kube-api-access-2flf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:20.988365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.984427 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" Oct 30 06:45:20.988772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:20.988747 5001 scope.go:122] "RemoveContainer" containerID="46a9a2d2436f13346046b1ded9780d3aa42d5b1b2ebd77b4a85b10c7fdbb1f0d" Oct 30 06:45:21.016396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.016255 5001 scope.go:122] "RemoveContainer" containerID="681d20e0c3b609dade73fab92c5d75896d87ea3a948f0c7ac76e6cc444201855" Oct 30 06:45:21.028088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.028059 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs"] Oct 30 06:45:21.041355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.041269 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad1661-29889045-9rgrs"] Oct 30 06:45:21.050446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.050269 5001 scope.go:122] "RemoveContainer" containerID="9568a64b4b715f2ce41b503bca31c4594eef3ccb46d4408cabd1b1e67ee6f397" Oct 30 06:45:21.051377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.050950 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-kube-api-access-nzpbn\" (UniqueName: \"kubernetes.io/projected/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-kube-api-access-nzpbn\") pod \"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08\" (UID: \"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08\") " Oct 30 06:45:21.054576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.054549 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-maas-api-service-ca\") pod \"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08\" (UID: \"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08\") " Oct 30 06:45:21.056405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.055099 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-maas-api-service-ca" pod "c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" (UID: "c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:21.056703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.056682 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2flf6\" (UniqueName: \"kubernetes.io/projected/50b32463-c8ed-417f-8718-c5ab4fd2f1b7-kube-api-access-2flf6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:21.056840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.056826 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:21.063029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.062911 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-kube-api-access-nzpbn" pod "c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" (UID: "c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08"). InnerVolumeSpecName "kube-api-access-nzpbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:21.084447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.084384 5001 scope.go:122] "RemoveContainer" containerID="3bb3da9cf0087186f2c2e86607d65514e3569b63d8750a2062e83c5424ae1c7d" Oct 30 06:45:21.110124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.110072 5001 scope.go:122] "RemoveContainer" containerID="d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb" Oct 30 06:45:21.119657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.119629 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" Oct 30 06:45:21.161357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.158392 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ced181d-88b9-407f-8663-01b2f28072c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ced181d-88b9-407f-8663-01b2f28072c2-maas-api-service-ca\") pod \"2ced181d-88b9-407f-8663-01b2f28072c2\" (UID: \"2ced181d-88b9-407f-8663-01b2f28072c2\") " Oct 30 06:45:21.161357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.158537 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ced181d-88b9-407f-8663-01b2f28072c2-kube-api-access-vvm6v\" (UniqueName: \"kubernetes.io/projected/2ced181d-88b9-407f-8663-01b2f28072c2-kube-api-access-vvm6v\") pod \"2ced181d-88b9-407f-8663-01b2f28072c2\" (UID: \"2ced181d-88b9-407f-8663-01b2f28072c2\") " Oct 30 06:45:21.161357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.159293 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzpbn\" (UniqueName: \"kubernetes.io/projected/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08-kube-api-access-nzpbn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:21.161357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.160021 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ced181d-88b9-407f-8663-01b2f28072c2-maas-api-service-ca" pod "2ced181d-88b9-407f-8663-01b2f28072c2" (UID: "2ced181d-88b9-407f-8663-01b2f28072c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:21.163190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.163161 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ced181d-88b9-407f-8663-01b2f28072c2-kube-api-access-vvm6v" pod "2ced181d-88b9-407f-8663-01b2f28072c2" (UID: "2ced181d-88b9-407f-8663-01b2f28072c2"). InnerVolumeSpecName "kube-api-access-vvm6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:21.223275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.223241 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw"] Oct 30 06:45:21.233023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.232993 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889045-2rgjw"] Oct 30 06:45:21.238610 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.238584 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:45:21.262387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.261924 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvm6v\" (UniqueName: \"kubernetes.io/projected/2ced181d-88b9-407f-8663-01b2f28072c2-kube-api-access-vvm6v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:21.262387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.261958 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ced181d-88b9-407f-8663-01b2f28072c2-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:21.262638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.262452 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x"] Oct 30 06:45:21.270901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.270676 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889045-ntr8x"] Oct 30 06:45:21.699261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.698938 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" event={"ID":"12f95efa-c8f1-4e3a-a5db-cf529d3cad30","Type":"ContainerStarted","Data":"70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb"} Oct 30 06:45:21.703934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.703895 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" Oct 30 06:45:21.703934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.703919 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq" event={"ID":"50b32463-c8ed-417f-8718-c5ab4fd2f1b7","Type":"ContainerDied","Data":"d4b3729e0f016c016de538e178fed946e5b3dc0fa0e1a0bc059b7c05eed83806"} Oct 30 06:45:21.704150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.703978 5001 scope.go:122] "RemoveContainer" containerID="7d1872f513de9d683ee960557b24b5976da095b0a69b2cba04a3d22bb734528d" Oct 30 06:45:21.707037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.707010 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk"] Oct 30 06:45:21.709365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.708896 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" event={"ID":"fe6534e6-dec1-42b8-b780-83abc658be3e","Type":"ContainerStarted","Data":"d81d6d70c0c389987dc228de86bf9be89c93743e88e924a9d75f402da6582113"} Oct 30 06:45:21.712995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.712910 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" event={"ID":"c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08","Type":"ContainerDied","Data":"3093bbeb57795fc4e3e34899deff11a4d3559d3e32f266fe0dae834a02508b35"} Oct 30 06:45:21.713125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.712999 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h" Oct 30 06:45:21.725500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.725469 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" event={"ID":"bb380de3-228b-448b-bc3b-848c1b638b11","Type":"ContainerStarted","Data":"c0e38ad2801b6c0e98e2fa53e727fb312c409f1655014e9554a1d5a900294bcf"} Oct 30 06:45:21.727648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.727508 5001 scope.go:122] "RemoveContainer" containerID="48f2ff4bc6e58547543f3efc6dc75d60a20dfeb13735bfe15554ccc28e318dd5" Oct 30 06:45:21.738890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.738540 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" event={"ID":"2ced181d-88b9-407f-8663-01b2f28072c2","Type":"ContainerDied","Data":"efd32b8d2dea2390c2416df4776ea541be570e790db6e5c0fa8d626e8a20a776"} Oct 30 06:45:21.738890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.738662 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf" Oct 30 06:45:21.743569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.743541 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8"] Oct 30 06:45:21.764598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.761508 5001 scope.go:122] "RemoveContainer" containerID="6b3701c1944c527f08908463c8a89154006377c3f3af5bd24b3c5d72165349dc" Oct 30 06:45:21.791521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.791056 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt"] Oct 30 06:45:21.908656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.906764 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw"] Oct 30 06:45:21.939472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.932849 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq"] Oct 30 06:45:21.954993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.954947 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69f3fa-29889045-dg7lq"] Oct 30 06:45:21.985219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.985157 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" podStartSLOduration=21.98513428 podStartE2EDuration="21.98513428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:21.966288774 +0000 UTC m=+3135.962245784" watchObservedRunningTime="2026-10-30 06:45:21.98513428 +0000 UTC m=+3135.981091290" Oct 30 06:45:21.998200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:21.998168 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl"] Oct 30 06:45:22.101626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.100138 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h"] Oct 30 06:45:22.102165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.102084 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" Oct 30 06:45:22.107469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.107443 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889045-st79h"] Oct 30 06:45:22.136299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.136158 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" path="/var/lib/kubelet/pods/0d40aa92-6ad3-4340-807b-b4e4796baa3d/volumes" Oct 30 06:45:22.149240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.149205 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" path="/var/lib/kubelet/pods/30471359-76f9-4579-be9a-4dd7d3af77a0/volumes" Oct 30 06:45:22.150170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.150147 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" path="/var/lib/kubelet/pods/50b32463-c8ed-417f-8718-c5ab4fd2f1b7/volumes" Oct 30 06:45:22.151273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.151255 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" path="/var/lib/kubelet/pods/62f0f671-aaa1-4a3f-a267-19d668b62f80/volumes" Oct 30 06:45:22.152046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.152028 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" path="/var/lib/kubelet/pods/c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08/volumes" Oct 30 06:45:22.202154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.202108 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" Oct 30 06:45:22.228849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.228819 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf"] Oct 30 06:45:22.296238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.294755 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc210264-c902-4f37-b47c-b25ce36a6482-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc210264-c902-4f37-b47c-b25ce36a6482-maas-api-service-ca\") pod \"fc210264-c902-4f37-b47c-b25ce36a6482\" (UID: \"fc210264-c902-4f37-b47c-b25ce36a6482\") " Oct 30 06:45:22.296238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.294905 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72dc0962-b2a8-479e-986a-e36ebdfc5e75-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72dc0962-b2a8-479e-986a-e36ebdfc5e75-maas-api-service-ca\") pod \"72dc0962-b2a8-479e-986a-e36ebdfc5e75\" (UID: \"72dc0962-b2a8-479e-986a-e36ebdfc5e75\") " Oct 30 06:45:22.296238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.295006 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72dc0962-b2a8-479e-986a-e36ebdfc5e75-kube-api-access-2p7k5\" (UniqueName: \"kubernetes.io/projected/72dc0962-b2a8-479e-986a-e36ebdfc5e75-kube-api-access-2p7k5\") pod \"72dc0962-b2a8-479e-986a-e36ebdfc5e75\" (UID: \"72dc0962-b2a8-479e-986a-e36ebdfc5e75\") " Oct 30 06:45:22.296238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.295184 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc210264-c902-4f37-b47c-b25ce36a6482-kube-api-access-7x685\" (UniqueName: \"kubernetes.io/projected/fc210264-c902-4f37-b47c-b25ce36a6482-kube-api-access-7x685\") pod \"fc210264-c902-4f37-b47c-b25ce36a6482\" (UID: \"fc210264-c902-4f37-b47c-b25ce36a6482\") " Oct 30 06:45:22.297549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.297234 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc210264-c902-4f37-b47c-b25ce36a6482-maas-api-service-ca" pod "fc210264-c902-4f37-b47c-b25ce36a6482" (UID: "fc210264-c902-4f37-b47c-b25ce36a6482"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:22.297838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.297797 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72dc0962-b2a8-479e-986a-e36ebdfc5e75-maas-api-service-ca" pod "72dc0962-b2a8-479e-986a-e36ebdfc5e75" (UID: "72dc0962-b2a8-479e-986a-e36ebdfc5e75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:22.367583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.367344 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88acce-29889045-6cvmf"] Oct 30 06:45:22.526332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.518607 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc210264-c902-4f37-b47c-b25ce36a6482-kube-api-access-7x685" pod "fc210264-c902-4f37-b47c-b25ce36a6482" (UID: "fc210264-c902-4f37-b47c-b25ce36a6482"). InnerVolumeSpecName "kube-api-access-7x685". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:22.557907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.540325 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72dc0962-b2a8-479e-986a-e36ebdfc5e75-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:22.557907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.540366 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x685\" (UniqueName: \"kubernetes.io/projected/fc210264-c902-4f37-b47c-b25ce36a6482-kube-api-access-7x685\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:22.557907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.540383 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc210264-c902-4f37-b47c-b25ce36a6482-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:22.830346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.830288 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72dc0962-b2a8-479e-986a-e36ebdfc5e75-kube-api-access-2p7k5" pod "72dc0962-b2a8-479e-986a-e36ebdfc5e75" (UID: "72dc0962-b2a8-479e-986a-e36ebdfc5e75"). InnerVolumeSpecName "kube-api-access-2p7k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:22.832874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.832843 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" event={"ID":"f74b178e-df08-4063-acdd-a0977b741211","Type":"ContainerDied","Data":"3a67876d6067e8af118e9839b5c8e27eea73cd70d455c0b9cd566886924a98bc"} Oct 30 06:45:22.833029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.832900 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a67876d6067e8af118e9839b5c8e27eea73cd70d455c0b9cd566886924a98bc" Oct 30 06:45:22.835117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.834769 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" Oct 30 06:45:22.835638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.835613 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" event={"ID":"fc210264-c902-4f37-b47c-b25ce36a6482","Type":"ContainerDied","Data":"392440bc035e47129e16d5a6df3dbeafa5be718dc1b3cfae41902287c3fb5b52"} Oct 30 06:45:22.835814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.835799 5001 scope.go:122] "RemoveContainer" containerID="91797fb66c05bffdb33d1883788c2d6043e6239d8c9d5df006a9b09c5695dcbc" Oct 30 06:45:22.836021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.836007 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8" Oct 30 06:45:22.888425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.888389 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" event={"ID":"72dc0962-b2a8-479e-986a-e36ebdfc5e75","Type":"ContainerDied","Data":"c4595101b6e5be7f61e3e28f14117003d38ea5e9ca939a7a51e91ca0b30bc6d0"} Oct 30 06:45:22.888789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.888762 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk" Oct 30 06:45:22.927410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.913934 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7"] Oct 30 06:45:22.927410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.917057 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p7k5\" (UniqueName: \"kubernetes.io/projected/72dc0962-b2a8-479e-986a-e36ebdfc5e75-kube-api-access-2p7k5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:22.927410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.918664 5001 generic.go:356] "Generic (PLEG): container finished" podID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerID="d81d6d70c0c389987dc228de86bf9be89c93743e88e924a9d75f402da6582113" exitCode=60 Oct 30 06:45:22.927410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.918838 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" event={"ID":"fe6534e6-dec1-42b8-b780-83abc658be3e","Type":"ContainerDied","Data":"d81d6d70c0c389987dc228de86bf9be89c93743e88e924a9d75f402da6582113"} Oct 30 06:45:22.927410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.923491 5001 generic.go:356] "Generic (PLEG): container finished" podID="bb380de3-228b-448b-bc3b-848c1b638b11" containerID="c0e38ad2801b6c0e98e2fa53e727fb312c409f1655014e9554a1d5a900294bcf" exitCode=60 Oct 30 06:45:22.927410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.923670 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerName="cleanup" containerID="cri-o://70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb" gracePeriod=30 Oct 30 06:45:22.927410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.924020 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" event={"ID":"bb380de3-228b-448b-bc3b-848c1b638b11","Type":"ContainerDied","Data":"c0e38ad2801b6c0e98e2fa53e727fb312c409f1655014e9554a1d5a900294bcf"} Oct 30 06:45:22.928733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.928176 5001 scope.go:122] "RemoveContainer" containerID="d81d6d70c0c389987dc228de86bf9be89c93743e88e924a9d75f402da6582113" Oct 30 06:45:22.933252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:22.932444 5001 scope.go:122] "RemoveContainer" containerID="c0e38ad2801b6c0e98e2fa53e727fb312c409f1655014e9554a1d5a900294bcf" Oct 30 06:45:22.933252 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:22.933013 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5_odh-ai-gateway-infra(bb380de3-228b-448b-bc3b-848c1b638b11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" Oct 30 06:45:22.937233 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:22.937193 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn_odh-ai-gateway-infra(fe6534e6-dec1-42b8-b780-83abc658be3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" Oct 30 06:45:23.019283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.019180 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74b178e-df08-4063-acdd-a0977b741211-kube-api-access-lkmpr\" (UniqueName: \"kubernetes.io/projected/f74b178e-df08-4063-acdd-a0977b741211-kube-api-access-lkmpr\") pod \"f74b178e-df08-4063-acdd-a0977b741211\" (UID: \"f74b178e-df08-4063-acdd-a0977b741211\") " Oct 30 06:45:23.019658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.019634 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74b178e-df08-4063-acdd-a0977b741211-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74b178e-df08-4063-acdd-a0977b741211-maas-api-service-ca\") pod \"f74b178e-df08-4063-acdd-a0977b741211\" (UID: \"f74b178e-df08-4063-acdd-a0977b741211\") " Oct 30 06:45:23.020800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.020776 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74b178e-df08-4063-acdd-a0977b741211-maas-api-service-ca" pod "f74b178e-df08-4063-acdd-a0977b741211" (UID: "f74b178e-df08-4063-acdd-a0977b741211"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:23.036548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.036514 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74b178e-df08-4063-acdd-a0977b741211-kube-api-access-lkmpr" pod "f74b178e-df08-4063-acdd-a0977b741211" (UID: "f74b178e-df08-4063-acdd-a0977b741211"). InnerVolumeSpecName "kube-api-access-lkmpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:23.043095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.043065 5001 scope.go:122] "RemoveContainer" containerID="a169eef7a7008a01441b879a9747932320c7c2ab818ac27f5071fd7080ea1528" Oct 30 06:45:23.070536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.064739 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn"] Oct 30 06:45:23.093194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.093069 5001 scope.go:122] "RemoveContainer" containerID="9bcce567bd8a81a49ed6de1a95cab4063c4a961b7eb82b8d497592cbfdc57638" Oct 30 06:45:23.096240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.095657 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" Oct 30 06:45:23.099210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.099189 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" Oct 30 06:45:23.105680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.105653 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5"] Oct 30 06:45:23.115592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.115109 5001 scope.go:122] "RemoveContainer" containerID="84d4f51e86c442d2f1fe2dc7f448cbaf95a464740395d59d17049f670322c228" Oct 30 06:45:23.122417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.122387 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkmpr\" (UniqueName: \"kubernetes.io/projected/f74b178e-df08-4063-acdd-a0977b741211-kube-api-access-lkmpr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:23.122545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.122422 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74b178e-df08-4063-acdd-a0977b741211-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:23.168646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.168566 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8"] Oct 30 06:45:23.174537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.174508 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f0145f-29889045-b4bd8"] Oct 30 06:45:23.223005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.222964 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70dfdbc8-cb44-4270-8108-11dddce181b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70dfdbc8-cb44-4270-8108-11dddce181b9-maas-api-service-ca\") pod \"70dfdbc8-cb44-4270-8108-11dddce181b9\" (UID: \"70dfdbc8-cb44-4270-8108-11dddce181b9\") " Oct 30 06:45:23.223175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.223081 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70dfdbc8-cb44-4270-8108-11dddce181b9-kube-api-access-kzzv7\" (UniqueName: \"kubernetes.io/projected/70dfdbc8-cb44-4270-8108-11dddce181b9-kube-api-access-kzzv7\") pod \"70dfdbc8-cb44-4270-8108-11dddce181b9\" (UID: \"70dfdbc8-cb44-4270-8108-11dddce181b9\") " Oct 30 06:45:23.223291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.223274 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/557b0e7c-2d56-49fc-913b-a4da8206101a-kube-api-access-6lvs8\" (UniqueName: \"kubernetes.io/projected/557b0e7c-2d56-49fc-913b-a4da8206101a-kube-api-access-6lvs8\") pod \"557b0e7c-2d56-49fc-913b-a4da8206101a\" (UID: \"557b0e7c-2d56-49fc-913b-a4da8206101a\") " Oct 30 06:45:23.223417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.223400 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/557b0e7c-2d56-49fc-913b-a4da8206101a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/557b0e7c-2d56-49fc-913b-a4da8206101a-maas-api-service-ca\") pod \"557b0e7c-2d56-49fc-913b-a4da8206101a\" (UID: \"557b0e7c-2d56-49fc-913b-a4da8206101a\") " Oct 30 06:45:23.225078 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.225046 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557b0e7c-2d56-49fc-913b-a4da8206101a-maas-api-service-ca" pod "557b0e7c-2d56-49fc-913b-a4da8206101a" (UID: "557b0e7c-2d56-49fc-913b-a4da8206101a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:23.225221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.225167 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70dfdbc8-cb44-4270-8108-11dddce181b9-maas-api-service-ca" pod "70dfdbc8-cb44-4270-8108-11dddce181b9" (UID: "70dfdbc8-cb44-4270-8108-11dddce181b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:23.227620 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.227596 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70dfdbc8-cb44-4270-8108-11dddce181b9-kube-api-access-kzzv7" pod "70dfdbc8-cb44-4270-8108-11dddce181b9" (UID: "70dfdbc8-cb44-4270-8108-11dddce181b9"). InnerVolumeSpecName "kube-api-access-kzzv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:23.227725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.227629 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557b0e7c-2d56-49fc-913b-a4da8206101a-kube-api-access-6lvs8" pod "557b0e7c-2d56-49fc-913b-a4da8206101a" (UID: "557b0e7c-2d56-49fc-913b-a4da8206101a"). InnerVolumeSpecName "kube-api-access-6lvs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:23.304852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.304814 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk"] Oct 30 06:45:23.306214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.306127 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" Oct 30 06:45:23.334550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.334509 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70dfdbc8-cb44-4270-8108-11dddce181b9-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:23.334550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.334549 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzzv7\" (UniqueName: \"kubernetes.io/projected/70dfdbc8-cb44-4270-8108-11dddce181b9-kube-api-access-kzzv7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:23.334776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.334572 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lvs8\" (UniqueName: \"kubernetes.io/projected/557b0e7c-2d56-49fc-913b-a4da8206101a-kube-api-access-6lvs8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:23.334776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.334590 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/557b0e7c-2d56-49fc-913b-a4da8206101a-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:23.340015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.339981 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c44730-29889045-hbfmk"] Oct 30 06:45:23.436031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.435925 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-kube-api-access-sb5bt\" (UniqueName: \"kubernetes.io/projected/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-kube-api-access-sb5bt\") pod \"12f95efa-c8f1-4e3a-a5db-cf529d3cad30\" (UID: \"12f95efa-c8f1-4e3a-a5db-cf529d3cad30\") " Oct 30 06:45:23.436184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.436156 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-maas-api-service-ca\") pod \"12f95efa-c8f1-4e3a-a5db-cf529d3cad30\" (UID: \"12f95efa-c8f1-4e3a-a5db-cf529d3cad30\") " Oct 30 06:45:23.436598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.436570 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-maas-api-service-ca" pod "12f95efa-c8f1-4e3a-a5db-cf529d3cad30" (UID: "12f95efa-c8f1-4e3a-a5db-cf529d3cad30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:23.437014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.436989 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:23.438949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.438924 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-kube-api-access-sb5bt" pod "12f95efa-c8f1-4e3a-a5db-cf529d3cad30" (UID: "12f95efa-c8f1-4e3a-a5db-cf529d3cad30"). InnerVolumeSpecName "kube-api-access-sb5bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:23.538744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.538708 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb5bt\" (UniqueName: \"kubernetes.io/projected/12f95efa-c8f1-4e3a-a5db-cf529d3cad30-kube-api-access-sb5bt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:23.934472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.934440 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" Oct 30 06:45:23.934974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.934443 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl" event={"ID":"557b0e7c-2d56-49fc-913b-a4da8206101a","Type":"ContainerDied","Data":"799558e1dfbdea739f3de94a7940af18ed8d50553f28e69131ec42601b360436"} Oct 30 06:45:23.934974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.934608 5001 scope.go:122] "RemoveContainer" containerID="0fa56e3acfdec1791e4ea18c5b70233be6d4eb9ace9e1e5e2a88ae113644516c" Oct 30 06:45:23.939676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.939643 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" event={"ID":"70dfdbc8-cb44-4270-8108-11dddce181b9","Type":"ContainerDied","Data":"6622a1d788d55af10873c80d3b0708c891111b54851cb69001bee2be6cb43a79"} Oct 30 06:45:23.939676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.939624 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw" Oct 30 06:45:23.941781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.941758 5001 generic.go:356] "Generic (PLEG): container finished" podID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerID="70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb" exitCode=60 Oct 30 06:45:23.941877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.941824 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" Oct 30 06:45:23.941877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.941836 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" event={"ID":"12f95efa-c8f1-4e3a-a5db-cf529d3cad30","Type":"ContainerDied","Data":"70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb"} Oct 30 06:45:23.942042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.941884 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7" event={"ID":"12f95efa-c8f1-4e3a-a5db-cf529d3cad30","Type":"ContainerDied","Data":"cec3d2655827a53d2b420d62c1c8ef75dc79c47b55efdd7b087d1067989cc34e"} Oct 30 06:45:23.948688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.948664 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt" Oct 30 06:45:23.949979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:23.949956 5001 scope.go:122] "RemoveContainer" containerID="0192b87a70135fada2c15aef621aca863a9f3f7ff7d18ec5c718e3520c290925" Oct 30 06:45:24.007423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.000567 5001 scope.go:122] "RemoveContainer" containerID="70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb" Oct 30 06:45:24.089436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.089395 5001 scope.go:122] "RemoveContainer" containerID="d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb" Oct 30 06:45:24.104980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.104959 5001 scope.go:122] "RemoveContainer" containerID="70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb" Oct 30 06:45:24.105478 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:24.105456 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb\": container with ID starting with 70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb not found: ID does not exist" containerID="70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb" Oct 30 06:45:24.105688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.105489 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb"} err="failed to get container status \"70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb\": rpc error: code = NotFound desc = could not find container \"70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb\": container with ID starting with 70269b08520d9d29d95a32712acedfd6f5489969f7e864c5340ad76f0a1187eb not found: ID does not exist" Oct 30 06:45:24.105688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.105508 5001 scope.go:122] "RemoveContainer" containerID="d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb" Oct 30 06:45:24.105811 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:24.105755 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb\": container with ID starting with d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb not found: ID does not exist" containerID="d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb" Oct 30 06:45:24.105811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.105778 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb"} err="failed to get container status \"d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb\": rpc error: code = NotFound desc = could not find container \"d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb\": container with ID starting with d7018b798d9e29c266590bd9c6cded62752d2b111e0d929034c78c7b09afbbfb not found: ID does not exist" Oct 30 06:45:24.159937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.159893 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" path="/var/lib/kubelet/pods/2ced181d-88b9-407f-8663-01b2f28072c2/volumes" Oct 30 06:45:24.160810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.160785 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" path="/var/lib/kubelet/pods/72dc0962-b2a8-479e-986a-e36ebdfc5e75/volumes" Oct 30 06:45:24.161619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.161598 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" path="/var/lib/kubelet/pods/fc210264-c902-4f37-b47c-b25ce36a6482/volumes" Oct 30 06:45:24.162813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.162792 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl"] Oct 30 06:45:24.162959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.162946 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889045-w8lpl"] Oct 30 06:45:24.186619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.186535 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt"] Oct 30 06:45:24.195512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.195466 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889045-25fxt"] Oct 30 06:45:24.242067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.242010 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw"] Oct 30 06:45:24.257503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.257470 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6d1778-29889045-d5dmw"] Oct 30 06:45:24.289042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.288971 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7"] Oct 30 06:45:24.305000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.298394 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a0083e-29889045-tmvq7"] Oct 30 06:45:24.325875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.325618 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" Oct 30 06:45:24.353551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.353522 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" Oct 30 06:45:24.362792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.362758 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb380de3-228b-448b-bc3b-848c1b638b11-kube-api-access-xrzt7\" (UniqueName: \"kubernetes.io/projected/bb380de3-228b-448b-bc3b-848c1b638b11-kube-api-access-xrzt7\") pod \"bb380de3-228b-448b-bc3b-848c1b638b11\" (UID: \"bb380de3-228b-448b-bc3b-848c1b638b11\") " Oct 30 06:45:24.362966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.362939 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe6534e6-dec1-42b8-b780-83abc658be3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe6534e6-dec1-42b8-b780-83abc658be3e-maas-api-service-ca\") pod \"fe6534e6-dec1-42b8-b780-83abc658be3e\" (UID: \"fe6534e6-dec1-42b8-b780-83abc658be3e\") " Oct 30 06:45:24.363052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.363022 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb380de3-228b-448b-bc3b-848c1b638b11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb380de3-228b-448b-bc3b-848c1b638b11-maas-api-service-ca\") pod \"bb380de3-228b-448b-bc3b-848c1b638b11\" (UID: \"bb380de3-228b-448b-bc3b-848c1b638b11\") " Oct 30 06:45:24.363359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.363130 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe6534e6-dec1-42b8-b780-83abc658be3e-kube-api-access-ghtpg\" (UniqueName: \"kubernetes.io/projected/fe6534e6-dec1-42b8-b780-83abc658be3e-kube-api-access-ghtpg\") pod \"fe6534e6-dec1-42b8-b780-83abc658be3e\" (UID: \"fe6534e6-dec1-42b8-b780-83abc658be3e\") " Oct 30 06:45:24.363637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.363512 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb380de3-228b-448b-bc3b-848c1b638b11-maas-api-service-ca" pod "bb380de3-228b-448b-bc3b-848c1b638b11" (UID: "bb380de3-228b-448b-bc3b-848c1b638b11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:24.363637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.363607 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6534e6-dec1-42b8-b780-83abc658be3e-maas-api-service-ca" pod "fe6534e6-dec1-42b8-b780-83abc658be3e" (UID: "fe6534e6-dec1-42b8-b780-83abc658be3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:24.364195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.364171 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe6534e6-dec1-42b8-b780-83abc658be3e-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:24.364260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.364203 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb380de3-228b-448b-bc3b-848c1b638b11-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:24.365782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.365755 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6534e6-dec1-42b8-b780-83abc658be3e-kube-api-access-ghtpg" pod "fe6534e6-dec1-42b8-b780-83abc658be3e" (UID: "fe6534e6-dec1-42b8-b780-83abc658be3e"). InnerVolumeSpecName "kube-api-access-ghtpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:24.366419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.365903 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb380de3-228b-448b-bc3b-848c1b638b11-kube-api-access-xrzt7" pod "bb380de3-228b-448b-bc3b-848c1b638b11" (UID: "bb380de3-228b-448b-bc3b-848c1b638b11"). InnerVolumeSpecName "kube-api-access-xrzt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:24.466573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.466451 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghtpg\" (UniqueName: \"kubernetes.io/projected/fe6534e6-dec1-42b8-b780-83abc658be3e-kube-api-access-ghtpg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:24.466573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.466491 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrzt7\" (UniqueName: \"kubernetes.io/projected/bb380de3-228b-448b-bc3b-848c1b638b11-kube-api-access-xrzt7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:45:24.963411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.963379 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" Oct 30 06:45:24.963919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.963422 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn" event={"ID":"fe6534e6-dec1-42b8-b780-83abc658be3e","Type":"ContainerDied","Data":"e65f194f3b79600daa7593c4ac45771a623465bbd92276a63ec6392114847534"} Oct 30 06:45:24.963919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.963488 5001 scope.go:122] "RemoveContainer" containerID="d81d6d70c0c389987dc228de86bf9be89c93743e88e924a9d75f402da6582113" Oct 30 06:45:24.966602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.966574 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" Oct 30 06:45:24.966740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.966598 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5" event={"ID":"bb380de3-228b-448b-bc3b-848c1b638b11","Type":"ContainerDied","Data":"e137411adb0e07e68e2a0a6ff6d42f4adf4a879b407bb6b613f9b62c70a4288a"} Oct 30 06:45:24.979940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:24.979920 5001 scope.go:122] "RemoveContainer" containerID="c0e38ad2801b6c0e98e2fa53e727fb312c409f1655014e9554a1d5a900294bcf" Oct 30 06:45:25.022742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:25.022702 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn"] Oct 30 06:45:25.041048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:25.041008 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d2923-29889045-hw7bn"] Oct 30 06:45:25.077786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:25.077172 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5"] Oct 30 06:45:25.084336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:25.084274 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889045-qsql5"] Oct 30 06:45:26.120491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:26.120460 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" path="/var/lib/kubelet/pods/12f95efa-c8f1-4e3a-a5db-cf529d3cad30/volumes" Oct 30 06:45:26.120952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:26.120939 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" path="/var/lib/kubelet/pods/557b0e7c-2d56-49fc-913b-a4da8206101a/volumes" Oct 30 06:45:26.121361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:26.121346 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" path="/var/lib/kubelet/pods/70dfdbc8-cb44-4270-8108-11dddce181b9/volumes" Oct 30 06:45:26.122049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:26.122032 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" path="/var/lib/kubelet/pods/bb380de3-228b-448b-bc3b-848c1b638b11/volumes" Oct 30 06:45:26.122499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:26.122483 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74b178e-df08-4063-acdd-a0977b741211" path="/var/lib/kubelet/pods/f74b178e-df08-4063-acdd-a0977b741211/volumes" Oct 30 06:45:26.122897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:26.122883 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" path="/var/lib/kubelet/pods/fe6534e6-dec1-42b8-b780-83abc658be3e/volumes" Oct 30 06:45:38.274733 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:45:38.274693 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:45:42.035767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:42.035730 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk"] Oct 30 06:45:42.048385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:42.048351 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce51c758-47pzk"] Oct 30 06:45:42.121506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:42.121475 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d58392-5b35-45c2-abd3-e8838e291c9f" path="/var/lib/kubelet/pods/85d58392-5b35-45c2-abd3-e8838e291c9f/volumes" Oct 30 06:45:52.035430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:52.035394 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc"] Oct 30 06:45:52.049313 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:52.049268 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03687e57-8kfhc"] Oct 30 06:45:52.119817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:45:52.119783 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c018948-98ed-4cbd-afd6-6f26e63908de" path="/var/lib/kubelet/pods/4c018948-98ed-4cbd-afd6-6f26e63908de/volumes" Oct 30 06:46:08.267652 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:46:08.267608 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:46:09.292794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:09.292753 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:46:11.214601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:11.214564 5001 scope.go:122] "RemoveContainer" containerID="323db2101c1dea379da8f307108200f1400662e8eaeb1fff8f36062173980ec3" Oct 30 06:46:11.229760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:11.229734 5001 scope.go:122] "RemoveContainer" containerID="d06cd38f8f1562292a9fffcb5ddca28ecb080ed9ca154c8352102b57ec0c4206" Oct 30 06:46:19.043436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:19.043346 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm"] Oct 30 06:46:19.056464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:19.056428 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcec73f4-j2fmm"] Oct 30 06:46:20.035496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:20.035448 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt"] Oct 30 06:46:20.048069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:20.048032 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c15d567-bpjnt"] Oct 30 06:46:20.120180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:20.120139 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663956cb-4785-4eed-b392-9764d9f52c20" path="/var/lib/kubelet/pods/663956cb-4785-4eed-b392-9764d9f52c20/volumes" Oct 30 06:46:20.120772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:20.120748 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6675be1-bb85-40d1-85ed-0ec1c78825c0" path="/var/lib/kubelet/pods/d6675be1-bb85-40d1-85ed-0ec1c78825c0/volumes" Oct 30 06:46:21.044007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:21.043969 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl"] Oct 30 06:46:21.056928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:21.056890 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4edf35b-fq6xl"] Oct 30 06:46:22.119897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:22.119864 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="301b496f-7f03-4b61-be50-9648294b3b3c" path="/var/lib/kubelet/pods/301b496f-7f03-4b61-be50-9648294b3b3c/volumes" Oct 30 06:46:32.036564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:32.036521 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj"] Oct 30 06:46:32.048289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:32.048258 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeabfa6f-npczj"] Oct 30 06:46:32.120758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:32.120723 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188927f8-697e-4b7f-9c6f-bd14ea0041f7" path="/var/lib/kubelet/pods/188927f8-697e-4b7f-9c6f-bd14ea0041f7/volumes" Oct 30 06:46:38.276128 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:46:38.276082 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:46:44.810819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:44.810780 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:46:53.111526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.111493 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9"] Oct 30 06:46:53.112316 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112285 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" containerName="cleanup" Oct 30 06:46:53.112375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112318 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" containerName="cleanup" Oct 30 06:46:53.112375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112334 5001 cpu_manager.go:414] "containerMap: removing container" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerName="cleanup" Oct 30 06:46:53.112375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112342 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerName="cleanup" Oct 30 06:46:53.112375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112355 5001 cpu_manager.go:414] "containerMap: removing container" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerName="cleanup" Oct 30 06:46:53.112375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112362 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112377 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112382 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112389 5001 cpu_manager.go:414] "containerMap: removing container" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112394 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112402 5001 cpu_manager.go:414] "containerMap: removing container" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112407 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112416 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112421 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112428 5001 cpu_manager.go:414] "containerMap: removing container" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112432 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112443 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112449 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112458 5001 cpu_manager.go:414] "containerMap: removing container" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112462 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112470 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112474 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112482 5001 cpu_manager.go:414] "containerMap: removing container" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112487 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112494 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112498 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112507 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112512 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112520 5001 cpu_manager.go:414] "containerMap: removing container" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112525 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112530 5001 cpu_manager.go:414] "containerMap: removing container" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112534 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112541 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f74b178e-df08-4063-acdd-a0977b741211" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112545 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74b178e-df08-4063-acdd-a0977b741211" containerName="cleanup" Oct 30 06:46:53.112540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112553 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112558 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112569 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f74b178e-df08-4063-acdd-a0977b741211" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112573 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74b178e-df08-4063-acdd-a0977b741211" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112578 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112582 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112589 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112594 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112603 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112607 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112613 5001 cpu_manager.go:414] "containerMap: removing container" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112617 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112623 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f74b178e-df08-4063-acdd-a0977b741211" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112627 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74b178e-df08-4063-acdd-a0977b741211" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112634 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a346576f-1151-4520-9dd7-dec9739f2032" containerName="payload-processing" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112638 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a346576f-1151-4520-9dd7-dec9739f2032" containerName="payload-processing" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112644 5001 cpu_manager.go:414] "containerMap: removing container" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112648 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112656 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112659 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112666 5001 cpu_manager.go:414] "containerMap: removing container" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112670 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112675 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112679 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112686 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112690 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112699 5001 cpu_manager.go:414] "containerMap: removing container" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112704 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112713 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112717 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112725 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112729 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112735 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112739 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112745 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112749 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112756 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112761 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112774 5001 cpu_manager.go:414] "containerMap: removing container" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112778 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112794 5001 cpu_manager.go:414] "containerMap: removing container" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112799 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112805 5001 cpu_manager.go:414] "containerMap: removing container" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112809 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112814 5001 cpu_manager.go:414] "containerMap: removing container" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112818 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112953 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112960 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112966 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74b178e-df08-4063-acdd-a0977b741211" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112973 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a346576f-1151-4520-9dd7-dec9739f2032" containerName="payload-processing" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112980 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112985 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112991 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.112995 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113001 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113007 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113011 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113017 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113021 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113028 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113033 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f71c0cf-a14f-4a6b-9067-9d60084a31c6" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113043 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113048 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113053 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113058 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113065 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b32463-c8ed-417f-8718-c5ab4fd2f1b7" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113072 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113078 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113083 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113088 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113094 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113098 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113103 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113109 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113115 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113119 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113127 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113132 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113139 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113145 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74b178e-df08-4063-acdd-a0977b741211" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113149 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerName="cleanup" Oct 30 06:46:53.113537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.113155 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb380de3-228b-448b-bc3b-848c1b638b11" containerName="cleanup" Oct 30 06:46:53.117428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.117412 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:53.136130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.136066 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9"] Oct 30 06:46:53.143092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.143062 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:46:53.143277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.143127 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:46:53.143277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.143190 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:46:53.148517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.148466 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:46:53.192827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.192781 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzcs5\" (UniqueName: \"kubernetes.io/projected/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-kube-api-access-jzcs5\") pod \"maas-api-revoke-sub-96617b2b-l9bw9\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:53.193041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.192934 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96617b2b-l9bw9\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:53.193136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.193119 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96617b2b-l9bw9\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:53.295049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.295010 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96617b2b-l9bw9\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:53.295333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.295280 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzcs5\" (UniqueName: \"kubernetes.io/projected/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-kube-api-access-jzcs5\") pod \"maas-api-revoke-sub-96617b2b-l9bw9\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:53.295430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.295412 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96617b2b-l9bw9\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:53.296098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.296074 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96617b2b-l9bw9\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:53.297688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.297669 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96617b2b-l9bw9\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:53.319870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.319840 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzcs5\" (UniqueName: \"kubernetes.io/projected/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-kube-api-access-jzcs5\") pod \"maas-api-revoke-sub-96617b2b-l9bw9\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:53.432838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.432756 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:53.490649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.490601 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v"] Oct 30 06:46:53.491998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.491974 5001 cpu_manager.go:414] "containerMap: removing container" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerName="cleanup" Oct 30 06:46:53.492123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.491999 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerName="cleanup" Oct 30 06:46:53.492123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492014 5001 cpu_manager.go:414] "containerMap: removing container" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerName="cleanup" Oct 30 06:46:53.492123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492022 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerName="cleanup" Oct 30 06:46:53.492123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492056 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" containerName="cleanup" Oct 30 06:46:53.492123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492064 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" containerName="cleanup" Oct 30 06:46:53.492123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492079 5001 cpu_manager.go:414] "containerMap: removing container" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerName="cleanup" Oct 30 06:46:53.492123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492086 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30471359-76f9-4579-be9a-4dd7d3af77a0" containerName="cleanup" Oct 30 06:46:53.492382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492127 5001 cpu_manager.go:414] "containerMap: removing container" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerName="cleanup" Oct 30 06:46:53.492382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492135 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerName="cleanup" Oct 30 06:46:53.492382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492178 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" containerName="cleanup" Oct 30 06:46:53.492382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492185 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc210264-c902-4f37-b47c-b25ce36a6482" containerName="cleanup" Oct 30 06:46:53.492529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492386 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d40aa92-6ad3-4340-807b-b4e4796baa3d" containerName="cleanup" Oct 30 06:46:53.492529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492411 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f0f671-aaa1-4a3f-a267-19d668b62f80" containerName="cleanup" Oct 30 06:46:53.492529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492422 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="70dfdbc8-cb44-4270-8108-11dddce181b9" containerName="cleanup" Oct 30 06:46:53.492529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492432 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="557b0e7c-2d56-49fc-913b-a4da8206101a" containerName="cleanup" Oct 30 06:46:53.492529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492443 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f95efa-c8f1-4e3a-a5db-cf529d3cad30" containerName="cleanup" Oct 30 06:46:53.492529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492461 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6534e6-dec1-42b8-b780-83abc658be3e" containerName="cleanup" Oct 30 06:46:53.492529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492471 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74b178e-df08-4063-acdd-a0977b741211" containerName="cleanup" Oct 30 06:46:53.492529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492480 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ced181d-88b9-407f-8663-01b2f28072c2" containerName="cleanup" Oct 30 06:46:53.492529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492490 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="72dc0962-b2a8-479e-986a-e36ebdfc5e75" containerName="cleanup" Oct 30 06:46:53.492529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.492501 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cd6f5b-7edc-4f70-8c40-8cbcde5a7e08" containerName="cleanup" Oct 30 06:46:53.498410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.498387 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.518460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.517722 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b66aad-openshift-default-dockercfg-tw72v\"" Oct 30 06:46:53.523079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.522652 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v"] Oct 30 06:46:53.701947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.629701 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istiod-ca-cert\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.732463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.732410 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6mb\" (UniqueName: \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-kube-api-access-fq6mb\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.732829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.732808 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-envoy\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.733091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.733075 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-certs\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.733256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.733243 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-podinfo\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.733411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.733392 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-socket\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.733549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.733533 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-token\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.733687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.733669 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-credential-socket\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.734021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.734000 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-data\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.879555 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-podinfo\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.879625 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-socket\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.879651 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-token\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.879683 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-credential-socket\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.879862 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-data\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.879957 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istiod-ca-cert\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.879995 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6mb\" (UniqueName: \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-kube-api-access-fq6mb\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.880109 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-envoy\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.880176 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-certs\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.880727 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-certs\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.910810 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-socket\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.911479 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-credential-socket\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.923151 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istiod-ca-cert\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:53.945334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:53.923291 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-data\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:54.003216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:54.001398 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-podinfo\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:54.012295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:54.008869 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-token\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:54.012295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:54.010744 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6mb\" (UniqueName: \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-kube-api-access-fq6mb\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:54.018365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:54.014288 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-envoy\") pod \"e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:54.141081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:54.141017 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:54.188805 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:46:54.188762 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500b90ce_a7a2_4aeb_bdeb_65f4eb79ecda.slice/crio-59436fc8a22a903ef528b499d17441387e75f102628113f31e7b263387da3266 WatchSource:0}: Error finding container 59436fc8a22a903ef528b499d17441387e75f102628113f31e7b263387da3266: Status 404 returned error can't find the container with id 59436fc8a22a903ef528b499d17441387e75f102628113f31e7b263387da3266 Oct 30 06:46:54.191575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:54.191543 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9"] Oct 30 06:46:54.482197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:54.482163 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v"] Oct 30 06:46:54.482469 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:46:54.482443 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edcda5b_e672_43ae_9fdf_10b0a644d0ae.slice/crio-a702796ef4ee99fd0ff4846f0c459e2967bafd6a4c4864015ccf19e7f11be073 WatchSource:0}: Error finding container a702796ef4ee99fd0ff4846f0c459e2967bafd6a4c4864015ccf19e7f11be073: Status 404 returned error can't find the container with id a702796ef4ee99fd0ff4846f0c459e2967bafd6a4c4864015ccf19e7f11be073 Oct 30 06:46:54.485965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:54.485931 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:46:54.486092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:54.486013 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:46:54.486092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:54.486058 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:46:55.052640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:55.052605 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" event={"ID":"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda","Type":"ContainerStarted","Data":"3a73613f87b620391c6ac4122ea2f12af3f026a1671a16b6f01eeb018dccf5f2"} Oct 30 06:46:55.052831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:55.052669 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" event={"ID":"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda","Type":"ContainerStarted","Data":"59436fc8a22a903ef528b499d17441387e75f102628113f31e7b263387da3266"} Oct 30 06:46:55.054827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:55.054794 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" event={"ID":"3edcda5b-e672-43ae-9fdf-10b0a644d0ae","Type":"ContainerStarted","Data":"60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a"} Oct 30 06:46:55.054967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:55.054849 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" event={"ID":"3edcda5b-e672-43ae-9fdf-10b0a644d0ae","Type":"ContainerStarted","Data":"a702796ef4ee99fd0ff4846f0c459e2967bafd6a4c4864015ccf19e7f11be073"} Oct 30 06:46:55.098748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:55.098700 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" podStartSLOduration=2.098685987 podStartE2EDuration="2.098685987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:46:55.095861801 +0000 UTC m=+3229.091818812" watchObservedRunningTime="2026-10-30 06:46:55.098685987 +0000 UTC m=+3229.094643001" Oct 30 06:46:55.141832 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:55.141735 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:55.145497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:55.145442 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" podStartSLOduration=2.145428696 podStartE2EDuration="2.145428696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:46:55.142056398 +0000 UTC m=+3229.138013410" watchObservedRunningTime="2026-10-30 06:46:55.145428696 +0000 UTC m=+3229.141385706" Oct 30 06:46:55.148293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:55.148269 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:56.069651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:56.069618 5001 generic.go:356] "Generic (PLEG): container finished" podID="500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda" containerID="3a73613f87b620391c6ac4122ea2f12af3f026a1671a16b6f01eeb018dccf5f2" exitCode=0 Oct 30 06:46:56.069864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:56.069701 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" event={"ID":"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda","Type":"ContainerDied","Data":"3a73613f87b620391c6ac4122ea2f12af3f026a1671a16b6f01eeb018dccf5f2"} Oct 30 06:46:56.070073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:56.070055 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:56.071130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:56.071108 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:46:57.316531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:57.316504 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:57.481524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:57.481491 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-kube-api-access-jzcs5\" (UniqueName: \"kubernetes.io/projected/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-kube-api-access-jzcs5\") pod \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " Oct 30 06:46:57.481747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:57.481668 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-openshift-ca\") pod \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " Oct 30 06:46:57.481856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:57.481840 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-xks-ca\") pod \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\" (UID: \"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda\") " Oct 30 06:46:57.482085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:57.482056 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-openshift-ca" pod "500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda" (UID: "500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:46:57.482527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:57.482505 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:46:57.483985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:57.483962 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-kube-api-access-jzcs5" pod "500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda" (UID: "500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda"). InnerVolumeSpecName "kube-api-access-jzcs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:46:57.484504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:57.484480 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-xks-ca" pod "500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda" (UID: "500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:46:57.584355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:57.584290 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:46:57.584355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:57.584353 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzcs5\" (UniqueName: \"kubernetes.io/projected/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda-kube-api-access-jzcs5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:46:58.091245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:58.091215 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" Oct 30 06:46:58.091245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:58.091235 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9" event={"ID":"500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda","Type":"ContainerDied","Data":"59436fc8a22a903ef528b499d17441387e75f102628113f31e7b263387da3266"} Oct 30 06:46:58.091549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:58.091368 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59436fc8a22a903ef528b499d17441387e75f102628113f31e7b263387da3266" Oct 30 06:46:59.554516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.554466 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb"] Oct 30 06:46:59.555786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.555760 5001 cpu_manager.go:414] "containerMap: removing container" podUID="500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda" containerName="revoke-keys" Oct 30 06:46:59.555786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.555786 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda" containerName="revoke-keys" Oct 30 06:46:59.556029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.556016 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda" containerName="revoke-keys" Oct 30 06:46:59.561061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.561039 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:46:59.580924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.580892 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb"] Oct 30 06:46:59.592328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.585563 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b66aad\"" Oct 30 06:46:59.592328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.585624 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b66aad-dockercfg-7f6j7\"" Oct 30 06:46:59.678336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.677385 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb\" (UID: \"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:46:59.678336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.677582 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm4sm\" (UniqueName: \"kubernetes.io/projected/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-kube-api-access-tm4sm\") pod \"payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb\" (UID: \"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:46:59.805395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.782385 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb\" (UID: \"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:46:59.805825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.805791 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm4sm\" (UniqueName: \"kubernetes.io/projected/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-kube-api-access-tm4sm\") pod \"payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb\" (UID: \"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:46:59.806465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.796838 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb\" (UID: \"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:46:59.879912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.879868 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm4sm\" (UniqueName: \"kubernetes.io/projected/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-kube-api-access-tm4sm\") pod \"payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb\" (UID: \"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:46:59.900692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.900653 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:46:59.953877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.953852 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj"] Oct 30 06:46:59.991022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.990986 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj"] Oct 30 06:46:59.991213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:46:59.991116 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:47:00.122527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:00.122484 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqksp\" (UniqueName: \"kubernetes.io/projected/c94e7069-6896-4127-b907-0a72f7b74da3-kube-api-access-zqksp\") pod \"payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj\" (UID: \"c94e7069-6896-4127-b907-0a72f7b74da3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:47:00.122947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:00.122918 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94e7069-6896-4127-b907-0a72f7b74da3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj\" (UID: \"c94e7069-6896-4127-b907-0a72f7b74da3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:47:00.224681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:00.224635 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94e7069-6896-4127-b907-0a72f7b74da3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj\" (UID: \"c94e7069-6896-4127-b907-0a72f7b74da3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:47:00.224883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:00.224713 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqksp\" (UniqueName: \"kubernetes.io/projected/c94e7069-6896-4127-b907-0a72f7b74da3-kube-api-access-zqksp\") pod \"payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj\" (UID: \"c94e7069-6896-4127-b907-0a72f7b74da3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:47:00.225470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:00.225447 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94e7069-6896-4127-b907-0a72f7b74da3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj\" (UID: \"c94e7069-6896-4127-b907-0a72f7b74da3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:47:00.249237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:00.249202 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqksp\" (UniqueName: \"kubernetes.io/projected/c94e7069-6896-4127-b907-0a72f7b74da3-kube-api-access-zqksp\") pod \"payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj\" (UID: \"c94e7069-6896-4127-b907-0a72f7b74da3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:47:00.322642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:00.322570 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb"] Oct 30 06:47:00.322819 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:47:00.322791 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d30c14_d3cf_46b1_8dd8_2f451f1bb8d6.slice/crio-92c88808847ad42e3d67c1a883951e909f7bdd1581b12d94bb35c488d6aee30c WatchSource:0}: Error finding container 92c88808847ad42e3d67c1a883951e909f7bdd1581b12d94bb35c488d6aee30c: Status 404 returned error can't find the container with id 92c88808847ad42e3d67c1a883951e909f7bdd1581b12d94bb35c488d6aee30c Oct 30 06:47:00.354132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:00.354105 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:47:00.714335 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:47:00.714287 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94e7069_6896_4127_b907_0a72f7b74da3.slice/crio-3f87d0b071badba583d2613a48ad29d2fc0f78c163a218855f7128653697d2cf WatchSource:0}: Error finding container 3f87d0b071badba583d2613a48ad29d2fc0f78c163a218855f7128653697d2cf: Status 404 returned error can't find the container with id 3f87d0b071badba583d2613a48ad29d2fc0f78c163a218855f7128653697d2cf Oct 30 06:47:00.714335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:00.714300 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj"] Oct 30 06:47:01.133182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:01.133136 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" event={"ID":"c94e7069-6896-4127-b907-0a72f7b74da3","Type":"ContainerStarted","Data":"ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f"} Oct 30 06:47:01.133470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:01.133236 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" event={"ID":"c94e7069-6896-4127-b907-0a72f7b74da3","Type":"ContainerStarted","Data":"3f87d0b071badba583d2613a48ad29d2fc0f78c163a218855f7128653697d2cf"} Oct 30 06:47:01.133470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:01.133440 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:47:01.134987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:01.134963 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" event={"ID":"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6","Type":"ContainerStarted","Data":"665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c"} Oct 30 06:47:01.135120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:01.135003 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" event={"ID":"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6","Type":"ContainerStarted","Data":"92c88808847ad42e3d67c1a883951e909f7bdd1581b12d94bb35c488d6aee30c"} Oct 30 06:47:01.135120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:01.135035 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:47:01.183572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:01.183504 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" podStartSLOduration=2.183483688 podStartE2EDuration="2.183483688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:01.177086622 +0000 UTC m=+3235.173043633" watchObservedRunningTime="2026-10-30 06:47:01.183483688 +0000 UTC m=+3235.179440699" Oct 30 06:47:01.222656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:01.222602 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" podStartSLOduration=2.222579774 podStartE2EDuration="2.222579774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:01.219032216 +0000 UTC m=+3235.214989226" watchObservedRunningTime="2026-10-30 06:47:01.222579774 +0000 UTC m=+3235.218536819" Oct 30 06:47:04.704350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.704270 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-dvrdh"] Oct 30 06:47:04.709422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.709395 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.733392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.733133 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-dvrdh"] Oct 30 06:47:04.757615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.757513 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-provider-credentials\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.758080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.758017 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktp99\" (UniqueName: \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-kube-api-access-ktp99\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.758373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.758352 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-routing-overlay-volume\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.758573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.758551 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-plugins-config-volume\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.860403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.860342 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktp99\" (UniqueName: \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-kube-api-access-ktp99\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.860626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.860442 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-routing-overlay-volume\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.860626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.860484 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-plugins-config-volume\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.860756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.860654 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-provider-credentials\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.861320 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.861272 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-routing-overlay-volume\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.861519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.861499 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-plugins-config-volume\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.863539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.863516 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-provider-credentials\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:04.885476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:04.885447 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktp99\" (UniqueName: \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-kube-api-access-ktp99\") pod \"payload-processing-external-model-578778c84f-dvrdh\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:05.026479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:05.026394 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:05.554062 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:05.554031 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-dvrdh"] Oct 30 06:47:05.557468 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:47:05.557443 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bafef1_f71c_45ac_a283_e06b56492180.slice/crio-d4640f4deaa4f69622fde18a9b7a09aaf5848651e1f6c292d00905bffee4879a WatchSource:0}: Error finding container d4640f4deaa4f69622fde18a9b7a09aaf5848651e1f6c292d00905bffee4879a: Status 404 returned error can't find the container with id d4640f4deaa4f69622fde18a9b7a09aaf5848651e1f6c292d00905bffee4879a Oct 30 06:47:06.433507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:06.433469 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" event={"ID":"d2bafef1-f71c-45ac-a283-e06b56492180","Type":"ContainerStarted","Data":"6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049"} Oct 30 06:47:06.434063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:06.433549 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" event={"ID":"d2bafef1-f71c-45ac-a283-e06b56492180","Type":"ContainerStarted","Data":"d4640f4deaa4f69622fde18a9b7a09aaf5848651e1f6c292d00905bffee4879a"} Oct 30 06:47:06.434063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:06.433717 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:06.482120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:06.482044 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" podStartSLOduration=2.482025042 podStartE2EDuration="2.482025042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:06.475965674 +0000 UTC m=+3240.471922688" watchObservedRunningTime="2026-10-30 06:47:06.482025042 +0000 UTC m=+3240.477982053" Oct 30 06:47:07.672432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.672395 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-dvrdh"] Oct 30 06:47:07.711183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.711151 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-86546bfd69-st8c4"] Oct 30 06:47:07.716217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.716191 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.740296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.740266 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86546bfd69-st8c4"] Oct 30 06:47:07.745623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.745595 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4c2398cafe36\"" Oct 30 06:47:07.765474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.765440 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-plugins-config-volume\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.765601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.765494 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-provider-credentials\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.765687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.765660 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-routing-overlay-volume\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.765896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.765860 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw4fk\" (UniqueName: \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-kube-api-access-nw4fk\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.867169 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.867137 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw4fk\" (UniqueName: \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-kube-api-access-nw4fk\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.867398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.867262 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-plugins-config-volume\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.867398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.867288 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-provider-credentials\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.867398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.867366 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-routing-overlay-volume\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.868039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.868003 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-routing-overlay-volume\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.868039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.868007 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-plugins-config-volume\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.870240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.870221 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-provider-credentials\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:07.891227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:07.891198 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw4fk\" (UniqueName: \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-kube-api-access-nw4fk\") pod \"payload-processing-external-model-86546bfd69-st8c4\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:08.032048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:08.031961 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:08.268871 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:08.268835 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:47:08.361412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:08.361377 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86546bfd69-st8c4"] Oct 30 06:47:08.361597 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:47:08.361465 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7dfcba_4f20_4b88_9f6d_d477d16316f5.slice/crio-989a9d48335523e574b47fcd58213b17622c6e9e7363db8dde0fdadc27067d2a WatchSource:0}: Error finding container 989a9d48335523e574b47fcd58213b17622c6e9e7363db8dde0fdadc27067d2a: Status 404 returned error can't find the container with id 989a9d48335523e574b47fcd58213b17622c6e9e7363db8dde0fdadc27067d2a Oct 30 06:47:08.472357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:08.472321 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" event={"ID":"6e7dfcba-4f20-4b88-9f6d-d477d16316f5","Type":"ContainerStarted","Data":"f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e"} Oct 30 06:47:08.472561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:08.472383 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" event={"ID":"6e7dfcba-4f20-4b88-9f6d-d477d16316f5","Type":"ContainerStarted","Data":"989a9d48335523e574b47fcd58213b17622c6e9e7363db8dde0fdadc27067d2a"} Oct 30 06:47:08.472561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:08.472416 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:08.523131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:08.523030 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" podStartSLOduration=1.523015663 podStartE2EDuration="1.523015663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:08.517055824 +0000 UTC m=+3242.513012833" watchObservedRunningTime="2026-10-30 06:47:08.523015663 +0000 UTC m=+3242.518972675" Oct 30 06:47:10.511384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:10.511339 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd"] Oct 30 06:47:10.511870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:10.511597 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" containerID="cri-o://285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe" gracePeriod=30 Oct 30 06:47:11.297843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.297809 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb"] Oct 30 06:47:11.303181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.303158 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:11.322032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.322007 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb"] Oct 30 06:47:11.329251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.329213 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:47:11.329408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.329255 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:47:11.329408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.329272 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:47:11.329408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.329357 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:47:11.407010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.406978 5001 scope.go:122] "RemoveContainer" containerID="6d6421e8f481c65db6d18e6af145d5cfbf9dee6b700be6925f7f56ab438d4571" Oct 30 06:47:11.418352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.418330 5001 scope.go:122] "RemoveContainer" containerID="68f498f84dca1e8b9447186a42da5b85a9fad8e377e4e18fa53be8852e70238d" Oct 30 06:47:11.423480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.423453 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87632f97-28gdb\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:11.423734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.423715 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjtfc\" (UniqueName: \"kubernetes.io/projected/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-kube-api-access-tjtfc\") pod \"maas-api-revoke-sub-87632f97-28gdb\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:11.423892 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.423871 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87632f97-28gdb\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:11.525515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.525479 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87632f97-28gdb\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:11.525981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.525570 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87632f97-28gdb\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:11.525981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.525722 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtfc\" (UniqueName: \"kubernetes.io/projected/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-kube-api-access-tjtfc\") pod \"maas-api-revoke-sub-87632f97-28gdb\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:11.526286 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.526264 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87632f97-28gdb\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:11.528071 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.528045 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87632f97-28gdb\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:11.550242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.550175 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjtfc\" (UniqueName: \"kubernetes.io/projected/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-kube-api-access-tjtfc\") pod \"maas-api-revoke-sub-87632f97-28gdb\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:11.624553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.624519 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:11.949732 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:47:11.949689 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4680fa3d_ad37_4552_bd5c_ea3ee7a2ad66.slice/crio-f4a4dc68b38baff88813b25353694ccae8ffc6ac6178c2451154f3a1683615f9 WatchSource:0}: Error finding container f4a4dc68b38baff88813b25353694ccae8ffc6ac6178c2451154f3a1683615f9: Status 404 returned error can't find the container with id f4a4dc68b38baff88813b25353694ccae8ffc6ac6178c2451154f3a1683615f9 Oct 30 06:47:11.950016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:11.949994 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb"] Oct 30 06:47:12.145967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:12.145939 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:47:12.146252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:12.146234 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:47:12.517401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:12.517358 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" event={"ID":"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66","Type":"ContainerStarted","Data":"2124aa42c74ce89ae4667a32c2ecee897ba8920998311c0bbb2dcc98fe2b8f8d"} Oct 30 06:47:12.517585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:12.517424 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" event={"ID":"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66","Type":"ContainerStarted","Data":"f4a4dc68b38baff88813b25353694ccae8ffc6ac6178c2451154f3a1683615f9"} Oct 30 06:47:12.655019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:12.654924 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" podStartSLOduration=1.654911558 podStartE2EDuration="1.654911558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:12.651944596 +0000 UTC m=+3246.647901608" watchObservedRunningTime="2026-10-30 06:47:12.654911558 +0000 UTC m=+3246.650868589" Oct 30 06:47:13.165857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.165816 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm"] Oct 30 06:47:13.166212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.166099 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" containerID="cri-o://4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10" gracePeriod=30 Oct 30 06:47:13.482333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.476489 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" podUID="d2bafef1-f71c-45ac-a283-e06b56492180" containerName="payload-processing" containerID="cri-o://6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049" gracePeriod=25 Oct 30 06:47:13.602902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.602854 5001 generic.go:356] "Generic (PLEG): container finished" podID="4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66" containerID="2124aa42c74ce89ae4667a32c2ecee897ba8920998311c0bbb2dcc98fe2b8f8d" exitCode=0 Oct 30 06:47:13.603263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.603236 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" event={"ID":"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66","Type":"ContainerDied","Data":"2124aa42c74ce89ae4667a32c2ecee897ba8920998311c0bbb2dcc98fe2b8f8d"} Oct 30 06:47:13.608247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.608215 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86546bfd69-st8c4"] Oct 30 06:47:13.613509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.613482 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:13.839349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.839281 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:13.878038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.876764 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-provider-credentials\") pod \"d2bafef1-f71c-45ac-a283-e06b56492180\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " Oct 30 06:47:13.878038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.876948 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-routing-overlay-volume\") pod \"d2bafef1-f71c-45ac-a283-e06b56492180\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " Oct 30 06:47:13.878038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.877209 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-plugins-config-volume\") pod \"d2bafef1-f71c-45ac-a283-e06b56492180\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " Oct 30 06:47:13.878038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.877257 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-kube-api-access-ktp99\" (UniqueName: \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-kube-api-access-ktp99\") pod \"d2bafef1-f71c-45ac-a283-e06b56492180\" (UID: \"d2bafef1-f71c-45ac-a283-e06b56492180\") " Oct 30 06:47:13.888342 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.885798 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-routing-overlay-volume" pod "d2bafef1-f71c-45ac-a283-e06b56492180" (UID: "d2bafef1-f71c-45ac-a283-e06b56492180"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:13.911778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.911722 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-provider-credentials" pod "d2bafef1-f71c-45ac-a283-e06b56492180" (UID: "d2bafef1-f71c-45ac-a283-e06b56492180"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:13.912548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.912496 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-plugins-config-volume" pod "d2bafef1-f71c-45ac-a283-e06b56492180" (UID: "d2bafef1-f71c-45ac-a283-e06b56492180"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:13.943577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.940565 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-kube-api-access-ktp99" pod "d2bafef1-f71c-45ac-a283-e06b56492180" (UID: "d2bafef1-f71c-45ac-a283-e06b56492180"). InnerVolumeSpecName "kube-api-access-ktp99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:13.968391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.968352 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq"] Oct 30 06:47:13.968962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.968907 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" containerID="cri-o://d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca" gracePeriod=30 Oct 30 06:47:13.997331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.984791 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:13.997331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.984829 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktp99\" (UniqueName: \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-kube-api-access-ktp99\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:13.997331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.984843 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2bafef1-f71c-45ac-a283-e06b56492180-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:13.997331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:13.984857 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2bafef1-f71c-45ac-a283-e06b56492180-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:14.334371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.334334 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:47:14.334541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.334401 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:47:14.615047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.615015 5001 generic.go:356] "Generic (PLEG): container finished" podID="d2bafef1-f71c-45ac-a283-e06b56492180" containerID="6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049" exitCode=0 Oct 30 06:47:14.615253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.615063 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" Oct 30 06:47:14.615253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.615100 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" event={"ID":"d2bafef1-f71c-45ac-a283-e06b56492180","Type":"ContainerDied","Data":"6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049"} Oct 30 06:47:14.615253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.615170 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-578778c84f-dvrdh" event={"ID":"d2bafef1-f71c-45ac-a283-e06b56492180","Type":"ContainerDied","Data":"d4640f4deaa4f69622fde18a9b7a09aaf5848651e1f6c292d00905bffee4879a"} Oct 30 06:47:14.615253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.615192 5001 scope.go:122] "RemoveContainer" containerID="6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049" Oct 30 06:47:14.627616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.627595 5001 scope.go:122] "RemoveContainer" containerID="6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049" Oct 30 06:47:14.627915 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:14.627885 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049\": container with ID starting with 6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049 not found: ID does not exist" containerID="6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049" Oct 30 06:47:14.628023 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.627922 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049"} err="failed to get container status \"6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049\": rpc error: code = NotFound desc = could not find container \"6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049\": container with ID starting with 6ad730d55cacdd9ebfae725401ca328c1a8a3044b526e5f74cb2ef33a3e70049 not found: ID does not exist" Oct 30 06:47:14.675845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.675814 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-dvrdh"] Oct 30 06:47:14.686765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.686731 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-dvrdh"] Oct 30 06:47:14.827133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.827099 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:14.902157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.902062 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-kube-api-access-tjtfc\" (UniqueName: \"kubernetes.io/projected/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-kube-api-access-tjtfc\") pod \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " Oct 30 06:47:14.902712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.902192 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-xks-ca\") pod \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " Oct 30 06:47:14.902712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.902373 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-openshift-ca\") pod \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\" (UID: \"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66\") " Oct 30 06:47:14.902886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.902858 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-openshift-ca" pod "4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66" (UID: "4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:14.903088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.903068 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:14.905190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.905161 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-xks-ca" pod "4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66" (UID: "4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:47:14.905658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:14.905629 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-kube-api-access-tjtfc" pod "4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66" (UID: "4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66"). InnerVolumeSpecName "kube-api-access-tjtfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:15.005080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.005042 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjtfc\" (UniqueName: \"kubernetes.io/projected/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-kube-api-access-tjtfc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:15.005080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.005078 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:15.633249 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.633223 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" Oct 30 06:47:15.633427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.633221 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb" event={"ID":"4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66","Type":"ContainerDied","Data":"f4a4dc68b38baff88813b25353694ccae8ffc6ac6178c2451154f3a1683615f9"} Oct 30 06:47:15.633427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.633369 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4a4dc68b38baff88813b25353694ccae8ffc6ac6178c2451154f3a1683615f9" Oct 30 06:47:15.842336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.842275 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:47:15.933520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.933483 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-data\") pod \"81d25535-ffbc-4239-abb4-9dca02a1b62e\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " Oct 30 06:47:15.934038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.933670 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-certs\") pod \"81d25535-ffbc-4239-abb4-9dca02a1b62e\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " Oct 30 06:47:15.934038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.933737 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-data" pod "81d25535-ffbc-4239-abb4-9dca02a1b62e" (UID: "81d25535-ffbc-4239-abb4-9dca02a1b62e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:15.934038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.933801 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-credential-socket\") pod \"81d25535-ffbc-4239-abb4-9dca02a1b62e\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " Oct 30 06:47:15.934038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.933829 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-envoy\") pod \"81d25535-ffbc-4239-abb4-9dca02a1b62e\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " Oct 30 06:47:15.934038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.933865 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-socket\") pod \"81d25535-ffbc-4239-abb4-9dca02a1b62e\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " Oct 30 06:47:15.934038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.933901 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d25535-ffbc-4239-abb4-9dca02a1b62e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d25535-ffbc-4239-abb4-9dca02a1b62e-istiod-ca-cert\") pod \"81d25535-ffbc-4239-abb4-9dca02a1b62e\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " Oct 30 06:47:15.934038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.933925 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-certs" pod "81d25535-ffbc-4239-abb4-9dca02a1b62e" (UID: "81d25535-ffbc-4239-abb4-9dca02a1b62e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:15.934038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.933932 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-podinfo\") pod \"81d25535-ffbc-4239-abb4-9dca02a1b62e\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " Oct 30 06:47:15.934600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.934110 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-credential-socket" pod "81d25535-ffbc-4239-abb4-9dca02a1b62e" (UID: "81d25535-ffbc-4239-abb4-9dca02a1b62e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:15.934600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.934143 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-token\" (UniqueName: \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-token\") pod \"81d25535-ffbc-4239-abb4-9dca02a1b62e\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " Oct 30 06:47:15.934600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.934195 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-kube-api-access-nlh7d\" (UniqueName: \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-kube-api-access-nlh7d\") pod \"81d25535-ffbc-4239-abb4-9dca02a1b62e\" (UID: \"81d25535-ffbc-4239-abb4-9dca02a1b62e\") " Oct 30 06:47:15.934600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.934208 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-socket" pod "81d25535-ffbc-4239-abb4-9dca02a1b62e" (UID: "81d25535-ffbc-4239-abb4-9dca02a1b62e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:15.934600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.934283 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d25535-ffbc-4239-abb4-9dca02a1b62e-istiod-ca-cert" pod "81d25535-ffbc-4239-abb4-9dca02a1b62e" (UID: "81d25535-ffbc-4239-abb4-9dca02a1b62e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:15.935231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.935207 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:15.935392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.935236 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:15.935392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.935255 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:15.935392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.935271 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d25535-ffbc-4239-abb4-9dca02a1b62e-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:15.935392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.935284 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:15.936680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.936649 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-token" pod "81d25535-ffbc-4239-abb4-9dca02a1b62e" (UID: "81d25535-ffbc-4239-abb4-9dca02a1b62e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:15.936778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.936690 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-kube-api-access-nlh7d" pod "81d25535-ffbc-4239-abb4-9dca02a1b62e" (UID: "81d25535-ffbc-4239-abb4-9dca02a1b62e"). InnerVolumeSpecName "kube-api-access-nlh7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:15.936905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.936872 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-podinfo" pod "81d25535-ffbc-4239-abb4-9dca02a1b62e" (UID: "81d25535-ffbc-4239-abb4-9dca02a1b62e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:15.937012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:15.936905 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-envoy" pod "81d25535-ffbc-4239-abb4-9dca02a1b62e" (UID: "81d25535-ffbc-4239-abb4-9dca02a1b62e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:16.036547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.036509 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:16.036547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.036542 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:16.036547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.036551 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:16.036547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.036559 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlh7d\" (UniqueName: \"kubernetes.io/projected/81d25535-ffbc-4239-abb4-9dca02a1b62e-kube-api-access-nlh7d\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:16.120092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.120055 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2bafef1-f71c-45ac-a283-e06b56492180" path="/var/lib/kubelet/pods/d2bafef1-f71c-45ac-a283-e06b56492180/volumes" Oct 30 06:47:16.643852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.643810 5001 generic.go:356] "Generic (PLEG): container finished" podID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerID="285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe" exitCode=0 Oct 30 06:47:16.644081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.643874 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" Oct 30 06:47:16.644081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.643892 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" event={"ID":"81d25535-ffbc-4239-abb4-9dca02a1b62e","Type":"ContainerDied","Data":"285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe"} Oct 30 06:47:16.644081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.643992 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd" event={"ID":"81d25535-ffbc-4239-abb4-9dca02a1b62e","Type":"ContainerDied","Data":"67e856dd63ec842f924480adb5a5d02c20cdcd3db4992e4b2138e16cf8cca853"} Oct 30 06:47:16.644081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.644025 5001 scope.go:122] "RemoveContainer" containerID="285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe" Oct 30 06:47:16.658005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.657984 5001 scope.go:122] "RemoveContainer" containerID="285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe" Oct 30 06:47:16.658487 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:16.658444 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe\": container with ID starting with 285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe not found: ID does not exist" containerID="285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe" Oct 30 06:47:16.658623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.658492 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe"} err="failed to get container status \"285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe\": rpc error: code = NotFound desc = could not find container \"285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe\": container with ID starting with 285ff0387bde4b1aee75e81a9a0e9feb6d65a55f401d36a77b2b59605aaa13fe not found: ID does not exist" Oct 30 06:47:16.715689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.715425 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd"] Oct 30 06:47:16.723851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.723811 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx"] Oct 30 06:47:16.725721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.725689 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66" containerName="revoke-keys" Oct 30 06:47:16.725893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.725733 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66" containerName="revoke-keys" Oct 30 06:47:16.725893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.725832 5001 cpu_manager.go:414] "containerMap: removing container" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" Oct 30 06:47:16.725893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.725842 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" Oct 30 06:47:16.725893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.725856 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d2bafef1-f71c-45ac-a283-e06b56492180" containerName="payload-processing" Oct 30 06:47:16.725893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.725863 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2bafef1-f71c-45ac-a283-e06b56492180" containerName="payload-processing" Oct 30 06:47:16.726190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.726151 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" containerName="istio-proxy" Oct 30 06:47:16.726190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.726169 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2bafef1-f71c-45ac-a283-e06b56492180" containerName="payload-processing" Oct 30 06:47:16.726321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.726209 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66" containerName="revoke-keys" Oct 30 06:47:16.732959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.732936 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-5gcmd"] Oct 30 06:47:16.733098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.733025 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.749398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.749369 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx"] Oct 30 06:47:16.750538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.750507 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-certs\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.750684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.750553 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-data\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.750741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.750684 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-podinfo\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.750842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.750817 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz7h7\" (UniqueName: \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-kube-api-access-mz7h7\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.751011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.750994 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-socket\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.751086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.751045 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-credential-socket\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.751171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.751154 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7e9f40-f353-436a-b771-ea6223ac02db-istiod-ca-cert\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.751222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.751196 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-token\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.751278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.751222 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-envoy\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.767373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.757237 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-62efb8-openshift-default-dockercfg-ts6p5\"" Oct 30 06:47:16.948384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.947416 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz7h7\" (UniqueName: \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-kube-api-access-mz7h7\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.948384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.947570 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-socket\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.948384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.947618 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-credential-socket\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.948384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.947656 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7e9f40-f353-436a-b771-ea6223ac02db-istiod-ca-cert\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.948384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.947705 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-token\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.948384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.947740 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-envoy\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.948384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.947863 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-certs\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.948384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.947890 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-data\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:16.948384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.947978 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-podinfo\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:17.010462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.966147 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7e9f40-f353-436a-b771-ea6223ac02db-istiod-ca-cert\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:17.010462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.966789 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-socket\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:17.010462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.967041 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-credential-socket\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:17.010462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.994014 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-certs\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:17.010462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.994340 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-data\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:17.010462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.997455 5001 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:47:17.010462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:16.997511 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:47:17.089360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:17.076277 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-podinfo\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:17.109329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:17.094003 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz7h7\" (UniqueName: \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-kube-api-access-mz7h7\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:17.109933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:17.109900 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-envoy\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:17.153143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:17.115229 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-token\") pod \"e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:17.351948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:17.351843 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:17.689907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:17.689879 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx"] Oct 30 06:47:17.691940 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:47:17.691906 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7e9f40_f353_436a_b771_ea6223ac02db.slice/crio-b20d91be833f271be2aa11e3358a7e2889da0e927e015e4f4b1eb51bcf420764 WatchSource:0}: Error finding container b20d91be833f271be2aa11e3358a7e2889da0e927e015e4f4b1eb51bcf420764: Status 404 returned error can't find the container with id b20d91be833f271be2aa11e3358a7e2889da0e927e015e4f4b1eb51bcf420764 Oct 30 06:47:17.695188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:17.695152 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:47:17.695284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:17.695221 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:47:17.695284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:17.695252 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:47:18.122253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.122219 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d25535-ffbc-4239-abb4-9dca02a1b62e" path="/var/lib/kubelet/pods/81d25535-ffbc-4239-abb4-9dca02a1b62e/volumes" Oct 30 06:47:18.493082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.493054 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:47:18.595423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.592543 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-kube-api-access-vtkmn\" (UniqueName: \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-kube-api-access-vtkmn\") pod \"37fe3b62-36bb-4285-805c-b88ccef37319\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " Oct 30 06:47:18.595423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.592734 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-envoy\") pod \"37fe3b62-36bb-4285-805c-b88ccef37319\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " Oct 30 06:47:18.595423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.592802 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-credential-socket\") pod \"37fe3b62-36bb-4285-805c-b88ccef37319\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " Oct 30 06:47:18.595423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.592841 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37fe3b62-36bb-4285-805c-b88ccef37319-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fe3b62-36bb-4285-805c-b88ccef37319-istio-podinfo\") pod \"37fe3b62-36bb-4285-805c-b88ccef37319\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " Oct 30 06:47:18.595423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.592905 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-istio-token\" (UniqueName: \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-istio-token\") pod \"37fe3b62-36bb-4285-805c-b88ccef37319\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " Oct 30 06:47:18.595423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.592983 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-data\") pod \"37fe3b62-36bb-4285-805c-b88ccef37319\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " Oct 30 06:47:18.595423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.593063 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-socket\") pod \"37fe3b62-36bb-4285-805c-b88ccef37319\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " Oct 30 06:47:18.595423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.593217 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-certs\") pod \"37fe3b62-36bb-4285-805c-b88ccef37319\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " Oct 30 06:47:18.595423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.593285 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fe3b62-36bb-4285-805c-b88ccef37319-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fe3b62-36bb-4285-805c-b88ccef37319-istiod-ca-cert\") pod \"37fe3b62-36bb-4285-805c-b88ccef37319\" (UID: \"37fe3b62-36bb-4285-805c-b88ccef37319\") " Oct 30 06:47:18.595423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.594380 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fe3b62-36bb-4285-805c-b88ccef37319-istiod-ca-cert" pod "37fe3b62-36bb-4285-805c-b88ccef37319" (UID: "37fe3b62-36bb-4285-805c-b88ccef37319"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:18.597072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.597043 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-credential-socket" pod "37fe3b62-36bb-4285-805c-b88ccef37319" (UID: "37fe3b62-36bb-4285-805c-b88ccef37319"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:18.604318 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.602846 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-socket" pod "37fe3b62-36bb-4285-805c-b88ccef37319" (UID: "37fe3b62-36bb-4285-805c-b88ccef37319"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:18.604318 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.603163 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-data" pod "37fe3b62-36bb-4285-805c-b88ccef37319" (UID: "37fe3b62-36bb-4285-805c-b88ccef37319"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:18.604318 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.603772 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-certs" pod "37fe3b62-36bb-4285-805c-b88ccef37319" (UID: "37fe3b62-36bb-4285-805c-b88ccef37319"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:18.605743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.605657 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-envoy" pod "37fe3b62-36bb-4285-805c-b88ccef37319" (UID: "37fe3b62-36bb-4285-805c-b88ccef37319"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:18.606221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.606184 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37fe3b62-36bb-4285-805c-b88ccef37319-istio-podinfo" pod "37fe3b62-36bb-4285-805c-b88ccef37319" (UID: "37fe3b62-36bb-4285-805c-b88ccef37319"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:18.609636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.609601 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" podUID="6e7dfcba-4f20-4b88-9f6d-d477d16316f5" containerName="payload-processing" containerID="cri-o://f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e" gracePeriod=25 Oct 30 06:47:18.618048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.617806 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-kube-api-access-vtkmn" pod "37fe3b62-36bb-4285-805c-b88ccef37319" (UID: "37fe3b62-36bb-4285-805c-b88ccef37319"). InnerVolumeSpecName "kube-api-access-vtkmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:18.620211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.620014 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-istio-token" pod "37fe3b62-36bb-4285-805c-b88ccef37319" (UID: "37fe3b62-36bb-4285-805c-b88ccef37319"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:18.684039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.683804 5001 generic.go:356] "Generic (PLEG): container finished" podID="37fe3b62-36bb-4285-805c-b88ccef37319" containerID="4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10" exitCode=0 Oct 30 06:47:18.684039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.683839 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" Oct 30 06:47:18.684039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.683861 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" event={"ID":"37fe3b62-36bb-4285-805c-b88ccef37319","Type":"ContainerDied","Data":"4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10"} Oct 30 06:47:18.684039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.683927 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm" event={"ID":"37fe3b62-36bb-4285-805c-b88ccef37319","Type":"ContainerDied","Data":"569d706458f63a4665c8681763dea5b0da4fce8caef2aaf89366ad7954a6decd"} Oct 30 06:47:18.684039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.683960 5001 scope.go:122] "RemoveContainer" containerID="4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10" Oct 30 06:47:18.686387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.686359 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" event={"ID":"0c7e9f40-f353-436a-b771-ea6223ac02db","Type":"ContainerStarted","Data":"cab136d2978bfa28648e8d879fe56ef5fc787f4c97ff1794c610f8ece6d28496"} Oct 30 06:47:18.686526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.686413 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" event={"ID":"0c7e9f40-f353-436a-b771-ea6223ac02db","Type":"ContainerStarted","Data":"b20d91be833f271be2aa11e3358a7e2889da0e927e015e4f4b1eb51bcf420764"} Oct 30 06:47:18.695833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.695809 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:18.695972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.695837 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fe3b62-36bb-4285-805c-b88ccef37319-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:18.695972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.695851 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtkmn\" (UniqueName: \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-kube-api-access-vtkmn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:18.695972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.695866 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:18.695972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.695881 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:18.695972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.695892 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fe3b62-36bb-4285-805c-b88ccef37319-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:18.695972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.695914 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fe3b62-36bb-4285-805c-b88ccef37319-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:18.695972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.695925 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:18.695972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.695938 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe3b62-36bb-4285-805c-b88ccef37319-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:18.727269 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.727114 5001 scope.go:122] "RemoveContainer" containerID="4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10" Oct 30 06:47:18.727588 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:18.727560 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10\": container with ID starting with 4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10 not found: ID does not exist" containerID="4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10" Oct 30 06:47:18.727684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.727604 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10"} err="failed to get container status \"4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10\": rpc error: code = NotFound desc = could not find container \"4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10\": container with ID starting with 4600644bb601168d704a9e1f7cc7c3ca982464a2a3e370339995bbcc046b9b10 not found: ID does not exist" Oct 30 06:47:18.750705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.750647 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" podStartSLOduration=2.750630588 podStartE2EDuration="2.750630588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:18.745837194 +0000 UTC m=+3252.741794206" watchObservedRunningTime="2026-10-30 06:47:18.750630588 +0000 UTC m=+3252.746587594" Oct 30 06:47:18.791742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.791703 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm"] Oct 30 06:47:18.807846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:18.807806 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8jslm"] Oct 30 06:47:19.024275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.024246 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:19.110108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.110061 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-plugins-config-volume\") pod \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " Oct 30 06:47:19.110470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.110445 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-plugins-config-volume" pod "6e7dfcba-4f20-4b88-9f6d-d477d16316f5" (UID: "6e7dfcba-4f20-4b88-9f6d-d477d16316f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:19.110542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.110516 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-kube-api-access-nw4fk\" (UniqueName: \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-kube-api-access-nw4fk\") pod \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " Oct 30 06:47:19.110631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.110616 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-routing-overlay-volume\") pod \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " Oct 30 06:47:19.110694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.110684 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-provider-credentials\") pod \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\" (UID: \"6e7dfcba-4f20-4b88-9f6d-d477d16316f5\") " Oct 30 06:47:19.111046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.111018 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-routing-overlay-volume" pod "6e7dfcba-4f20-4b88-9f6d-d477d16316f5" (UID: "6e7dfcba-4f20-4b88-9f6d-d477d16316f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:19.111366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.111343 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.111366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.111366 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.112781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.112755 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-kube-api-access-nw4fk" pod "6e7dfcba-4f20-4b88-9f6d-d477d16316f5" (UID: "6e7dfcba-4f20-4b88-9f6d-d477d16316f5"). InnerVolumeSpecName "kube-api-access-nw4fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:19.113228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.113203 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-provider-credentials" pod "6e7dfcba-4f20-4b88-9f6d-d477d16316f5" (UID: "6e7dfcba-4f20-4b88-9f6d-d477d16316f5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:19.212860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.212831 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.212860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.212859 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw4fk\" (UniqueName: \"kubernetes.io/projected/6e7dfcba-4f20-4b88-9f6d-d477d16316f5-kube-api-access-nw4fk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.353219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.353144 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:19.370558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.361416 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:19.442866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.442838 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:47:19.618124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618079 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc10096-6be7-4413-bb12-684879f00809-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc10096-6be7-4413-bb12-684879f00809-istiod-ca-cert\") pod \"bcc10096-6be7-4413-bb12-684879f00809\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " Oct 30 06:47:19.618354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618178 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-istio-token\" (UniqueName: \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-istio-token\") pod \"bcc10096-6be7-4413-bb12-684879f00809\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " Oct 30 06:47:19.618354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618245 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-certs\") pod \"bcc10096-6be7-4413-bb12-684879f00809\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " Oct 30 06:47:19.618354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618279 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-data\") pod \"bcc10096-6be7-4413-bb12-684879f00809\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " Oct 30 06:47:19.618354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618324 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-socket\") pod \"bcc10096-6be7-4413-bb12-684879f00809\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " Oct 30 06:47:19.618636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618425 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcc10096-6be7-4413-bb12-684879f00809-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc10096-6be7-4413-bb12-684879f00809-istio-podinfo\") pod \"bcc10096-6be7-4413-bb12-684879f00809\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " Oct 30 06:47:19.618636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618455 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-kube-api-access-pdm85\" (UniqueName: \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-kube-api-access-pdm85\") pod \"bcc10096-6be7-4413-bb12-684879f00809\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " Oct 30 06:47:19.618636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618522 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc10096-6be7-4413-bb12-684879f00809-istiod-ca-cert" pod "bcc10096-6be7-4413-bb12-684879f00809" (UID: "bcc10096-6be7-4413-bb12-684879f00809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:19.618853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618671 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-socket" pod "bcc10096-6be7-4413-bb12-684879f00809" (UID: "bcc10096-6be7-4413-bb12-684879f00809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:19.618853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618689 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-certs" pod "bcc10096-6be7-4413-bb12-684879f00809" (UID: "bcc10096-6be7-4413-bb12-684879f00809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:19.618853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618812 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-credential-socket\") pod \"bcc10096-6be7-4413-bb12-684879f00809\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " Oct 30 06:47:19.619056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618858 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-envoy\") pod \"bcc10096-6be7-4413-bb12-684879f00809\" (UID: \"bcc10096-6be7-4413-bb12-684879f00809\") " Oct 30 06:47:19.619056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.618904 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-data" pod "bcc10096-6be7-4413-bb12-684879f00809" (UID: "bcc10096-6be7-4413-bb12-684879f00809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:19.619173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.619107 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-credential-socket" pod "bcc10096-6be7-4413-bb12-684879f00809" (UID: "bcc10096-6be7-4413-bb12-684879f00809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:19.619634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.619610 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.619747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.619638 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.619747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.619651 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.619747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.619663 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.619747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.619677 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc10096-6be7-4413-bb12-684879f00809-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.621120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.621093 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcc10096-6be7-4413-bb12-684879f00809-istio-podinfo" pod "bcc10096-6be7-4413-bb12-684879f00809" (UID: "bcc10096-6be7-4413-bb12-684879f00809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:19.621477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.621450 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-istio-token" pod "bcc10096-6be7-4413-bb12-684879f00809" (UID: "bcc10096-6be7-4413-bb12-684879f00809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:19.621587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.621499 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-kube-api-access-pdm85" pod "bcc10096-6be7-4413-bb12-684879f00809" (UID: "bcc10096-6be7-4413-bb12-684879f00809"). InnerVolumeSpecName "kube-api-access-pdm85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:19.621712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.621692 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-envoy" pod "bcc10096-6be7-4413-bb12-684879f00809" (UID: "bcc10096-6be7-4413-bb12-684879f00809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:19.702057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.702022 5001 generic.go:356] "Generic (PLEG): container finished" podID="bcc10096-6be7-4413-bb12-684879f00809" containerID="d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca" exitCode=0 Oct 30 06:47:19.702239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.702077 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" Oct 30 06:47:19.702239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.702073 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" event={"ID":"bcc10096-6be7-4413-bb12-684879f00809","Type":"ContainerDied","Data":"d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca"} Oct 30 06:47:19.702427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.702257 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq" event={"ID":"bcc10096-6be7-4413-bb12-684879f00809","Type":"ContainerDied","Data":"6db7413688324966f7e6f1e948be87601d72a1b1151bc3f023e5841071169aa3"} Oct 30 06:47:19.702427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.702295 5001 scope.go:122] "RemoveContainer" containerID="d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca" Oct 30 06:47:19.704444 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.704415 5001 generic.go:356] "Generic (PLEG): container finished" podID="6e7dfcba-4f20-4b88-9f6d-d477d16316f5" containerID="f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e" exitCode=0 Oct 30 06:47:19.704738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.704707 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" Oct 30 06:47:19.705564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.705532 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" event={"ID":"6e7dfcba-4f20-4b88-9f6d-d477d16316f5","Type":"ContainerDied","Data":"f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e"} Oct 30 06:47:19.705695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.705586 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:19.705695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.705612 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86546bfd69-st8c4" event={"ID":"6e7dfcba-4f20-4b88-9f6d-d477d16316f5","Type":"ContainerDied","Data":"989a9d48335523e574b47fcd58213b17622c6e9e7363db8dde0fdadc27067d2a"} Oct 30 06:47:19.707396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.707372 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:47:19.719417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.719398 5001 scope.go:122] "RemoveContainer" containerID="d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca" Oct 30 06:47:19.719726 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:19.719707 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca\": container with ID starting with d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca not found: ID does not exist" containerID="d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca" Oct 30 06:47:19.719813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.719738 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca"} err="failed to get container status \"d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca\": rpc error: code = NotFound desc = could not find container \"d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca\": container with ID starting with d46bd6f3d62aa6d0bc689590f8871c130c3dc8fa5d76e6b5a3c7fe9cb79952ca not found: ID does not exist" Oct 30 06:47:19.719813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.719761 5001 scope.go:122] "RemoveContainer" containerID="f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e" Oct 30 06:47:19.721117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.721094 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc10096-6be7-4413-bb12-684879f00809-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.721175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.721125 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdm85\" (UniqueName: \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-kube-api-access-pdm85\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.721175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.721141 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc10096-6be7-4413-bb12-684879f00809-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.721175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.721154 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc10096-6be7-4413-bb12-684879f00809-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:19.731656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.731454 5001 scope.go:122] "RemoveContainer" containerID="f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e" Oct 30 06:47:19.731791 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:19.731771 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e\": container with ID starting with f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e not found: ID does not exist" containerID="f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e" Oct 30 06:47:19.731833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.731801 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e"} err="failed to get container status \"f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e\": rpc error: code = NotFound desc = could not find container \"f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e\": container with ID starting with f04ed0de82dc5266aa4da523ac6db0f0d0d42ddf185a346e9ed62ae505d8769e not found: ID does not exist" Oct 30 06:47:19.834280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.834231 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq"] Oct 30 06:47:19.839284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.839245 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-cx5nq"] Oct 30 06:47:19.867562 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.867526 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86546bfd69-st8c4"] Oct 30 06:47:19.872570 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:19.872501 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-86546bfd69-st8c4"] Oct 30 06:47:20.121686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:20.121656 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" path="/var/lib/kubelet/pods/37fe3b62-36bb-4285-805c-b88ccef37319/volumes" Oct 30 06:47:20.122194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:20.122178 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e7dfcba-4f20-4b88-9f6d-d477d16316f5" path="/var/lib/kubelet/pods/6e7dfcba-4f20-4b88-9f6d-d477d16316f5/volumes" Oct 30 06:47:20.122737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:20.122684 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc10096-6be7-4413-bb12-684879f00809" path="/var/lib/kubelet/pods/bcc10096-6be7-4413-bb12-684879f00809/volumes" Oct 30 06:47:22.337554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.337515 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4"] Oct 30 06:47:22.338606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.338568 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6e7dfcba-4f20-4b88-9f6d-d477d16316f5" containerName="payload-processing" Oct 30 06:47:22.338606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.338604 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7dfcba-4f20-4b88-9f6d-d477d16316f5" containerName="payload-processing" Oct 30 06:47:22.338810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.338624 5001 cpu_manager.go:414] "containerMap: removing container" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" Oct 30 06:47:22.338810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.338631 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" Oct 30 06:47:22.338810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.338680 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" Oct 30 06:47:22.338810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.338689 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" Oct 30 06:47:22.339084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.338940 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7dfcba-4f20-4b88-9f6d-d477d16316f5" containerName="payload-processing" Oct 30 06:47:22.339084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.338957 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fe3b62-36bb-4285-805c-b88ccef37319" containerName="istio-proxy" Oct 30 06:47:22.339084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.338973 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc10096-6be7-4413-bb12-684879f00809" containerName="istio-proxy" Oct 30 06:47:22.344826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.344805 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:22.361343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.361274 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4"] Oct 30 06:47:22.368696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.368671 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:47:22.368838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.368671 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:47:22.375059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.374477 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:47:22.375059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.374584 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:47:22.378594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.378573 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmnbt\" (UniqueName: \"kubernetes.io/projected/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-kube-api-access-fmnbt\") pod \"maas-api-revoke-sub-f0c70f51-lmpr4\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:22.378715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.378625 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0c70f51-lmpr4\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:22.378839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.378819 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0c70f51-lmpr4\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:22.480552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.480516 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmnbt\" (UniqueName: \"kubernetes.io/projected/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-kube-api-access-fmnbt\") pod \"maas-api-revoke-sub-f0c70f51-lmpr4\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:22.480782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.480607 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0c70f51-lmpr4\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:22.480873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.480786 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0c70f51-lmpr4\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:22.481836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.481730 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0c70f51-lmpr4\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:22.484076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.484045 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0c70f51-lmpr4\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:22.504967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.504938 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmnbt\" (UniqueName: \"kubernetes.io/projected/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-kube-api-access-fmnbt\") pod \"maas-api-revoke-sub-f0c70f51-lmpr4\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:22.666111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.666079 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:22.993459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:22.993410 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4"] Oct 30 06:47:23.006096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.006029 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2"] Oct 30 06:47:23.016081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.016052 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:47:23.032226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.032201 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-62efb8\"" Oct 30 06:47:23.038710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.035114 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2"] Oct 30 06:47:23.075053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.075017 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf"] Oct 30 06:47:23.086487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.086458 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:47:23.098230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.098201 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf"] Oct 30 06:47:23.099769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.099740 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsft4\" (UniqueName: \"kubernetes.io/projected/d781ce9b-5bf3-4ea9-870b-b22a77d02929-kube-api-access-jsft4\") pod \"payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2\" (UID: \"d781ce9b-5bf3-4ea9-870b-b22a77d02929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:47:23.099933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.099916 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d781ce9b-5bf3-4ea9-870b-b22a77d02929-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2\" (UID: \"d781ce9b-5bf3-4ea9-870b-b22a77d02929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:47:23.100003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.099983 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05977063-0120-460c-bee1-8ae684ea6ac7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf\" (UID: \"05977063-0120-460c-bee1-8ae684ea6ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:47:23.100077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.100023 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdck\" (UniqueName: \"kubernetes.io/projected/05977063-0120-460c-bee1-8ae684ea6ac7-kube-api-access-dzdck\") pod \"payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf\" (UID: \"05977063-0120-460c-bee1-8ae684ea6ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:47:23.112349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.112287 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-62efb8-dockercfg-rcc5h\"" Oct 30 06:47:23.209333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.205998 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d781ce9b-5bf3-4ea9-870b-b22a77d02929-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2\" (UID: \"d781ce9b-5bf3-4ea9-870b-b22a77d02929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:47:23.235796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.235694 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05977063-0120-460c-bee1-8ae684ea6ac7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf\" (UID: \"05977063-0120-460c-bee1-8ae684ea6ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:47:23.235796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.235769 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdck\" (UniqueName: \"kubernetes.io/projected/05977063-0120-460c-bee1-8ae684ea6ac7-kube-api-access-dzdck\") pod \"payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf\" (UID: \"05977063-0120-460c-bee1-8ae684ea6ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:47:23.236095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.236068 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsft4\" (UniqueName: \"kubernetes.io/projected/d781ce9b-5bf3-4ea9-870b-b22a77d02929-kube-api-access-jsft4\") pod \"payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2\" (UID: \"d781ce9b-5bf3-4ea9-870b-b22a77d02929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:47:23.236696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.229619 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d781ce9b-5bf3-4ea9-870b-b22a77d02929-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2\" (UID: \"d781ce9b-5bf3-4ea9-870b-b22a77d02929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:47:23.242326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.237536 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05977063-0120-460c-bee1-8ae684ea6ac7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf\" (UID: \"05977063-0120-460c-bee1-8ae684ea6ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:47:23.430817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.412461 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdck\" (UniqueName: \"kubernetes.io/projected/05977063-0120-460c-bee1-8ae684ea6ac7-kube-api-access-dzdck\") pod \"payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf\" (UID: \"05977063-0120-460c-bee1-8ae684ea6ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:47:23.430817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.416927 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsft4\" (UniqueName: \"kubernetes.io/projected/d781ce9b-5bf3-4ea9-870b-b22a77d02929-kube-api-access-jsft4\") pod \"payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2\" (UID: \"d781ce9b-5bf3-4ea9-870b-b22a77d02929\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:47:23.485955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.485849 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:47:23.675972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.675936 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:47:23.762628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.762591 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" event={"ID":"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d","Type":"ContainerStarted","Data":"cca24a341b0f60df1ad73f690e85392ac0c328cbee9bd14febc715cd4fcc7e52"} Oct 30 06:47:23.762790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.762656 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" event={"ID":"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d","Type":"ContainerStarted","Data":"3bcca4c785c7df7c4cf54430c8f828d273d3a5dd01005e53177b75a8d72be786"} Oct 30 06:47:23.809287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.809216 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf"] Oct 30 06:47:23.816229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:23.816170 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" podStartSLOduration=1.816150528 podStartE2EDuration="1.816150528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:23.805956972 +0000 UTC m=+3257.801913982" watchObservedRunningTime="2026-10-30 06:47:23.816150528 +0000 UTC m=+3257.812107540" Oct 30 06:47:24.055928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:24.055874 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2"] Oct 30 06:47:24.056936 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:47:24.056903 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd781ce9b_5bf3_4ea9_870b_b22a77d02929.slice/crio-868edd90db6467ef52a95a6ab955cec2df24ab6f939336f07d1d8e1593abeb1c WatchSource:0}: Error finding container 868edd90db6467ef52a95a6ab955cec2df24ab6f939336f07d1d8e1593abeb1c: Status 404 returned error can't find the container with id 868edd90db6467ef52a95a6ab955cec2df24ab6f939336f07d1d8e1593abeb1c Oct 30 06:47:24.773067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:24.772977 5001 generic.go:356] "Generic (PLEG): container finished" podID="84b8e92e-ca86-4a1d-ada3-f99d7f5de94d" containerID="cca24a341b0f60df1ad73f690e85392ac0c328cbee9bd14febc715cd4fcc7e52" exitCode=0 Oct 30 06:47:24.773574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:24.773080 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" event={"ID":"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d","Type":"ContainerDied","Data":"cca24a341b0f60df1ad73f690e85392ac0c328cbee9bd14febc715cd4fcc7e52"} Oct 30 06:47:24.774972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:24.774946 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" event={"ID":"d781ce9b-5bf3-4ea9-870b-b22a77d02929","Type":"ContainerStarted","Data":"102fef86ee7a0a308e7978dbc2de26e64539b0e9cd5318575dbefa75b70d60d0"} Oct 30 06:47:24.775099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:24.774990 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" event={"ID":"d781ce9b-5bf3-4ea9-870b-b22a77d02929","Type":"ContainerStarted","Data":"868edd90db6467ef52a95a6ab955cec2df24ab6f939336f07d1d8e1593abeb1c"} Oct 30 06:47:24.775099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:24.775015 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:47:24.776882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:24.776856 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" event={"ID":"05977063-0120-460c-bee1-8ae684ea6ac7","Type":"ContainerStarted","Data":"59343c3b3a13fb39f6703363d8c4181383d4934c5d5df6da989cc46fd303e8e3"} Oct 30 06:47:24.776985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:24.776904 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" event={"ID":"05977063-0120-460c-bee1-8ae684ea6ac7","Type":"ContainerStarted","Data":"7d2c60b52227c23c6746898f118a475d709ece3c79a0147538ce8e3b59432b83"} Oct 30 06:47:24.776985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:24.776933 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:47:24.861091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:24.861032 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" podStartSLOduration=1.861012051 podStartE2EDuration="1.861012051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:24.859935395 +0000 UTC m=+3258.855892407" watchObservedRunningTime="2026-10-30 06:47:24.861012051 +0000 UTC m=+3258.856969063" Oct 30 06:47:24.904293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:24.904225 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" podStartSLOduration=2.904205337 podStartE2EDuration="2.904205337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:24.902513547 +0000 UTC m=+3258.898470558" watchObservedRunningTime="2026-10-30 06:47:24.904205337 +0000 UTC m=+3258.900162350" Oct 30 06:47:25.561008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:25.560966 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766"] Oct 30 06:47:25.561236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:25.561212 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerName="istio-proxy" containerID="cri-o://28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93" gracePeriod=30 Oct 30 06:47:26.033675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.033647 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:26.113319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.113283 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-openshift-ca\") pod \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " Oct 30 06:47:26.113544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.113403 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-xks-ca\") pod \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " Oct 30 06:47:26.113665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.113645 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-kube-api-access-fmnbt\" (UniqueName: \"kubernetes.io/projected/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-kube-api-access-fmnbt\") pod \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\" (UID: \"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d\") " Oct 30 06:47:26.113760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.113736 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-openshift-ca" pod "84b8e92e-ca86-4a1d-ada3-f99d7f5de94d" (UID: "84b8e92e-ca86-4a1d-ada3-f99d7f5de94d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:26.114989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.114964 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:26.116276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.116251 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-kube-api-access-fmnbt" pod "84b8e92e-ca86-4a1d-ada3-f99d7f5de94d" (UID: "84b8e92e-ca86-4a1d-ada3-f99d7f5de94d"). InnerVolumeSpecName "kube-api-access-fmnbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:26.116467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.116447 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-xks-ca" pod "84b8e92e-ca86-4a1d-ada3-f99d7f5de94d" (UID: "84b8e92e-ca86-4a1d-ada3-f99d7f5de94d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:47:26.216475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.216441 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:26.216475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.216474 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmnbt\" (UniqueName: \"kubernetes.io/projected/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d-kube-api-access-fmnbt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:26.228053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.228021 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj"] Oct 30 06:47:26.835788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.835760 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" Oct 30 06:47:26.835788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.835768 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4" event={"ID":"84b8e92e-ca86-4a1d-ada3-f99d7f5de94d","Type":"ContainerDied","Data":"3bcca4c785c7df7c4cf54430c8f828d273d3a5dd01005e53177b75a8d72be786"} Oct 30 06:47:26.836067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:26.835817 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bcca4c785c7df7c4cf54430c8f828d273d3a5dd01005e53177b75a8d72be786" Oct 30 06:47:27.936866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:27.936828 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg"] Oct 30 06:47:27.937406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:27.937096 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" containerID="cri-o://d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9" gracePeriod=30 Oct 30 06:47:27.949715 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:27.949681 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn"] Oct 30 06:47:27.949909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:27.949875 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" podUID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" containerName="istio-proxy" containerID="cri-o://dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f" gracePeriod=30 Oct 30 06:47:30.865454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:30.865424 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:47:30.877380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:30.877344 5001 generic.go:356] "Generic (PLEG): container finished" podID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerID="28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93" exitCode=0 Oct 30 06:47:30.877551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:30.877392 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" Oct 30 06:47:30.877551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:30.877406 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" event={"ID":"99a380d4-e404-41ed-9e49-1af4f8b822d8","Type":"ContainerDied","Data":"28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93"} Oct 30 06:47:30.877551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:30.877467 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766" event={"ID":"99a380d4-e404-41ed-9e49-1af4f8b822d8","Type":"ContainerDied","Data":"a430d19edd7a16edd08352a460cd45555df6b61b136ea855684d0771f27446a5"} Oct 30 06:47:30.877551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:30.877500 5001 scope.go:122] "RemoveContainer" containerID="28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93" Oct 30 06:47:30.889967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:30.889942 5001 scope.go:122] "RemoveContainer" containerID="28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93" Oct 30 06:47:30.890287 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:30.890265 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93\": container with ID starting with 28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93 not found: ID does not exist" containerID="28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93" Oct 30 06:47:30.890381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:30.890296 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93"} err="failed to get container status \"28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93\": rpc error: code = NotFound desc = could not find container \"28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93\": container with ID starting with 28acd58ed53369252491bc13d5933bcb47cc5ffcb331fb750463de75a2e51e93 not found: ID does not exist" Oct 30 06:47:31.001911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.001869 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a380d4-e404-41ed-9e49-1af4f8b822d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a380d4-e404-41ed-9e49-1af4f8b822d8-istiod-ca-cert\") pod \"99a380d4-e404-41ed-9e49-1af4f8b822d8\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " Oct 30 06:47:31.002125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.001966 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-credential-socket\") pod \"99a380d4-e404-41ed-9e49-1af4f8b822d8\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " Oct 30 06:47:31.002125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002059 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-podinfo\") pod \"99a380d4-e404-41ed-9e49-1af4f8b822d8\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " Oct 30 06:47:31.002224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002188 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-certs\") pod \"99a380d4-e404-41ed-9e49-1af4f8b822d8\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " Oct 30 06:47:31.002541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002319 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-token\" (UniqueName: \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-token\") pod \"99a380d4-e404-41ed-9e49-1af4f8b822d8\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " Oct 30 06:47:31.002541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002356 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-envoy\") pod \"99a380d4-e404-41ed-9e49-1af4f8b822d8\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " Oct 30 06:47:31.002541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002391 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a380d4-e404-41ed-9e49-1af4f8b822d8-istiod-ca-cert" pod "99a380d4-e404-41ed-9e49-1af4f8b822d8" (UID: "99a380d4-e404-41ed-9e49-1af4f8b822d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:31.002541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002373 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-credential-socket" pod "99a380d4-e404-41ed-9e49-1af4f8b822d8" (UID: "99a380d4-e404-41ed-9e49-1af4f8b822d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:31.002541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002439 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-socket\") pod \"99a380d4-e404-41ed-9e49-1af4f8b822d8\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " Oct 30 06:47:31.002541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002451 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-certs" pod "99a380d4-e404-41ed-9e49-1af4f8b822d8" (UID: "99a380d4-e404-41ed-9e49-1af4f8b822d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:31.002541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002474 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-kube-api-access-klj49\" (UniqueName: \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-kube-api-access-klj49\") pod \"99a380d4-e404-41ed-9e49-1af4f8b822d8\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " Oct 30 06:47:31.002541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002540 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-data\") pod \"99a380d4-e404-41ed-9e49-1af4f8b822d8\" (UID: \"99a380d4-e404-41ed-9e49-1af4f8b822d8\") " Oct 30 06:47:31.003035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002693 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-socket" pod "99a380d4-e404-41ed-9e49-1af4f8b822d8" (UID: "99a380d4-e404-41ed-9e49-1af4f8b822d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:31.003035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.002976 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-data" pod "99a380d4-e404-41ed-9e49-1af4f8b822d8" (UID: "99a380d4-e404-41ed-9e49-1af4f8b822d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:31.003404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.003375 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.003404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.003398 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.003539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.003412 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.003539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.003424 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a380d4-e404-41ed-9e49-1af4f8b822d8-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.003539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.003435 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.004892 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.004866 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-token" pod "99a380d4-e404-41ed-9e49-1af4f8b822d8" (UID: "99a380d4-e404-41ed-9e49-1af4f8b822d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:31.005129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.005115 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-podinfo" pod "99a380d4-e404-41ed-9e49-1af4f8b822d8" (UID: "99a380d4-e404-41ed-9e49-1af4f8b822d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:31.005446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.005422 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-envoy" pod "99a380d4-e404-41ed-9e49-1af4f8b822d8" (UID: "99a380d4-e404-41ed-9e49-1af4f8b822d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:31.005548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.005444 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-kube-api-access-klj49" pod "99a380d4-e404-41ed-9e49-1af4f8b822d8" (UID: "99a380d4-e404-41ed-9e49-1af4f8b822d8"). InnerVolumeSpecName "kube-api-access-klj49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:31.105253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.105217 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.105253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.105254 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.105483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.105264 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klj49\" (UniqueName: \"kubernetes.io/projected/99a380d4-e404-41ed-9e49-1af4f8b822d8-kube-api-access-klj49\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.105483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.105277 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a380d4-e404-41ed-9e49-1af4f8b822d8-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.228740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.228698 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" podUID="121c8afc-303a-449b-81dc-8c3e795b598d" containerName="payload-processing" containerID="cri-o://198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f" gracePeriod=25 Oct 30 06:47:31.230405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.230367 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766"] Oct 30 06:47:31.242116 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.242080 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-hg766"] Oct 30 06:47:31.550486 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.550451 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:47:31.613602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.613565 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-kube-api-access-64sn9\" (UniqueName: \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-kube-api-access-64sn9\") pod \"121c8afc-303a-449b-81dc-8c3e795b598d\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " Oct 30 06:47:31.613818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.613684 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-plugins-config-volume\") pod \"121c8afc-303a-449b-81dc-8c3e795b598d\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " Oct 30 06:47:31.613818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.613737 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-routing-overlay-volume\") pod \"121c8afc-303a-449b-81dc-8c3e795b598d\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " Oct 30 06:47:31.613952 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.613824 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-provider-credentials\") pod \"121c8afc-303a-449b-81dc-8c3e795b598d\" (UID: \"121c8afc-303a-449b-81dc-8c3e795b598d\") " Oct 30 06:47:31.615691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.615660 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-routing-overlay-volume" pod "121c8afc-303a-449b-81dc-8c3e795b598d" (UID: "121c8afc-303a-449b-81dc-8c3e795b598d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:31.615883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.615864 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-plugins-config-volume" pod "121c8afc-303a-449b-81dc-8c3e795b598d" (UID: "121c8afc-303a-449b-81dc-8c3e795b598d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:31.619582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.619553 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-provider-credentials" pod "121c8afc-303a-449b-81dc-8c3e795b598d" (UID: "121c8afc-303a-449b-81dc-8c3e795b598d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:31.622498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.622449 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-kube-api-access-64sn9" pod "121c8afc-303a-449b-81dc-8c3e795b598d" (UID: "121c8afc-303a-449b-81dc-8c3e795b598d"). InnerVolumeSpecName "kube-api-access-64sn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:31.719111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.719072 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.719111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.719110 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64sn9\" (UniqueName: \"kubernetes.io/projected/121c8afc-303a-449b-81dc-8c3e795b598d-kube-api-access-64sn9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.719548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.719126 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.719548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.719141 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/121c8afc-303a-449b-81dc-8c3e795b598d-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.891379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.890713 5001 generic.go:356] "Generic (PLEG): container finished" podID="121c8afc-303a-449b-81dc-8c3e795b598d" containerID="198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f" exitCode=0 Oct 30 06:47:31.891379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.890821 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" Oct 30 06:47:31.891379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.890982 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" event={"ID":"121c8afc-303a-449b-81dc-8c3e795b598d","Type":"ContainerDied","Data":"198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f"} Oct 30 06:47:31.891992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.891884 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj" event={"ID":"121c8afc-303a-449b-81dc-8c3e795b598d","Type":"ContainerDied","Data":"a578de998caab64e0a43fc134994bcccf02c867b9fa32abb945dd5d3d49f866c"} Oct 30 06:47:31.891992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.891940 5001 scope.go:122] "RemoveContainer" containerID="198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f" Oct 30 06:47:31.907298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.907271 5001 scope.go:122] "RemoveContainer" containerID="198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f" Oct 30 06:47:31.907611 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:31.907592 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f\": container with ID starting with 198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f not found: ID does not exist" containerID="198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f" Oct 30 06:47:31.907704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.907619 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f"} err="failed to get container status \"198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f\": rpc error: code = NotFound desc = could not find container \"198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f\": container with ID starting with 198b30ceaba0281f66ad8e4a82b17bb9f79054ef673e87cd3f915c852490725f not found: ID does not exist" Oct 30 06:47:31.950872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.950816 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj"] Oct 30 06:47:31.957739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:31.957698 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65dff8f9bc-kxmrj"] Oct 30 06:47:32.120413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:32.120377 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121c8afc-303a-449b-81dc-8c3e795b598d" path="/var/lib/kubelet/pods/121c8afc-303a-449b-81dc-8c3e795b598d/volumes" Oct 30 06:47:32.120927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:32.120912 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" path="/var/lib/kubelet/pods/99a380d4-e404-41ed-9e49-1af4f8b822d8/volumes" Oct 30 06:47:33.378296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.378265 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:47:33.406833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.406806 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:47:33.454849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.454806 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-podinfo\") pod \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " Oct 30 06:47:33.455032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.454869 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-token\" (UniqueName: \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-token\") pod \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " Oct 30 06:47:33.455032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.454938 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-socket\") pod \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " Oct 30 06:47:33.455975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.455943 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-data\") pod \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " Oct 30 06:47:33.456356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.456330 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istiod-ca-cert\") pod \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " Oct 30 06:47:33.456502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.456486 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-credential-socket\") pod \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " Oct 30 06:47:33.456723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.456707 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-envoy\") pod \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " Oct 30 06:47:33.456907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.456893 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-certs\") pod \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " Oct 30 06:47:33.457047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.457032 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-kube-api-access-s4r65\" (UniqueName: \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-kube-api-access-s4r65\") pod \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\" (UID: \"b2bf08e0-cac0-43ed-847a-3df6d9003e3c\") " Oct 30 06:47:33.457498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.457451 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-socket" pod "b2bf08e0-cac0-43ed-847a-3df6d9003e3c" (UID: "b2bf08e0-cac0-43ed-847a-3df6d9003e3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.458701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.458675 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.459758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.459356 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-data" pod "b2bf08e0-cac0-43ed-847a-3df6d9003e3c" (UID: "b2bf08e0-cac0-43ed-847a-3df6d9003e3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.459917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.459708 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istiod-ca-cert" pod "b2bf08e0-cac0-43ed-847a-3df6d9003e3c" (UID: "b2bf08e0-cac0-43ed-847a-3df6d9003e3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:33.460271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.460237 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-credential-socket" pod "b2bf08e0-cac0-43ed-847a-3df6d9003e3c" (UID: "b2bf08e0-cac0-43ed-847a-3df6d9003e3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.460510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.460485 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-certs" pod "b2bf08e0-cac0-43ed-847a-3df6d9003e3c" (UID: "b2bf08e0-cac0-43ed-847a-3df6d9003e3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.463326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.463241 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-kube-api-access-s4r65" pod "b2bf08e0-cac0-43ed-847a-3df6d9003e3c" (UID: "b2bf08e0-cac0-43ed-847a-3df6d9003e3c"). InnerVolumeSpecName "kube-api-access-s4r65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:33.463851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.463811 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-token" pod "b2bf08e0-cac0-43ed-847a-3df6d9003e3c" (UID: "b2bf08e0-cac0-43ed-847a-3df6d9003e3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:33.474460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.473970 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-podinfo" pod "b2bf08e0-cac0-43ed-847a-3df6d9003e3c" (UID: "b2bf08e0-cac0-43ed-847a-3df6d9003e3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:33.474460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.474233 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-envoy" pod "b2bf08e0-cac0-43ed-847a-3df6d9003e3c" (UID: "b2bf08e0-cac0-43ed-847a-3df6d9003e3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.560709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.560666 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-envoy\") pod \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " Oct 30 06:47:33.560900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.560754 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-data\") pod \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " Oct 30 06:47:33.561079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.561049 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-data" pod "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" (UID: "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.561289 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.561268 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-certs\") pod \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " Oct 30 06:47:33.561663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.561641 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istiod-ca-cert\") pod \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " Oct 30 06:47:33.561843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.561825 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-credential-socket\") pod \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " Oct 30 06:47:33.561985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.561967 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-kube-api-access-8r2n6\" (UniqueName: \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-kube-api-access-8r2n6\") pod \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " Oct 30 06:47:33.562108 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.562094 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-podinfo\") pod \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " Oct 30 06:47:33.562238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.562223 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-socket\") pod \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " Oct 30 06:47:33.562572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.562549 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-token\" (UniqueName: \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-token\") pod \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\" (UID: \"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415\") " Oct 30 06:47:33.563498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.563472 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.563624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.563610 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.563741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.563727 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.563841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.563827 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.563949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.563936 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4r65\" (UniqueName: \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-kube-api-access-s4r65\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.564037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.564024 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.565661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.565646 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.567785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.567769 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.567940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.567927 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bf08e0-cac0-43ed-847a-3df6d9003e3c-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.568204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.564291 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-envoy" pod "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" (UID: "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.568354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.564534 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istiod-ca-cert" pod "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" (UID: "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:33.568457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.565074 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-socket" pod "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" (UID: "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.568558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.565585 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-certs" pod "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" (UID: "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.568717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.567126 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-podinfo" pod "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" (UID: "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:33.568823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.567719 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-token" pod "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" (UID: "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:33.568904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.568674 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-credential-socket" pod "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" (UID: "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.569293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.569269 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-kube-api-access-8r2n6" pod "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" (UID: "f5fc25a2-998b-4a99-8ab2-ac5d4b11f415"). InnerVolumeSpecName "kube-api-access-8r2n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:33.669995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.669960 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.669995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.669993 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.669995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.670005 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.670275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.670019 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.670275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.670034 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r2n6\" (UniqueName: \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-kube-api-access-8r2n6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.670275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.670046 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.670275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.670057 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.670275 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.670066 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.925874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.925841 5001 generic.go:356] "Generic (PLEG): container finished" podID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" containerID="dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f" exitCode=0 Oct 30 06:47:33.926051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.925919 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" Oct 30 06:47:33.926051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.925929 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" event={"ID":"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415","Type":"ContainerDied","Data":"dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f"} Oct 30 06:47:33.926051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.926015 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn" event={"ID":"f5fc25a2-998b-4a99-8ab2-ac5d4b11f415","Type":"ContainerDied","Data":"f3c411b77d3b62e1fb88b374c03f6a263e161d2b8a0602b0cd751ca0fd2678bb"} Oct 30 06:47:33.926051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.926035 5001 scope.go:122] "RemoveContainer" containerID="dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f" Oct 30 06:47:33.928410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.928382 5001 generic.go:356] "Generic (PLEG): container finished" podID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerID="d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9" exitCode=0 Oct 30 06:47:33.928545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.928456 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" Oct 30 06:47:33.928545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.928460 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" event={"ID":"b2bf08e0-cac0-43ed-847a-3df6d9003e3c","Type":"ContainerDied","Data":"d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9"} Oct 30 06:47:33.928545 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.928521 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg" event={"ID":"b2bf08e0-cac0-43ed-847a-3df6d9003e3c","Type":"ContainerDied","Data":"2b2c970903384201e91e1e1860ad331f9b680d8c33dab3aa025cf5de4b761437"} Oct 30 06:47:33.949220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.948666 5001 scope.go:122] "RemoveContainer" containerID="dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f" Oct 30 06:47:33.952860 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:33.952431 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f\": container with ID starting with dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f not found: ID does not exist" containerID="dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f" Oct 30 06:47:33.952860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.952478 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f"} err="failed to get container status \"dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f\": rpc error: code = NotFound desc = could not find container \"dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f\": container with ID starting with dfff1daef0e14538a62d289fef0bf4ce0342fc40e88a2dbef775bcd5cc4a0b3f not found: ID does not exist" Oct 30 06:47:33.952860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.952503 5001 scope.go:122] "RemoveContainer" containerID="d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9" Oct 30 06:47:33.972221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.972174 5001 scope.go:122] "RemoveContainer" containerID="5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f" Oct 30 06:47:33.999003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.998978 5001 scope.go:122] "RemoveContainer" containerID="d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9" Oct 30 06:47:33.999390 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:33.999364 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9\": container with ID starting with d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9 not found: ID does not exist" containerID="d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9" Oct 30 06:47:33.999518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.999396 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9"} err="failed to get container status \"d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9\": rpc error: code = NotFound desc = could not find container \"d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9\": container with ID starting with d214b9075a6b90236897a6e89cc08052cd6c44985c79464b99bc4613600b47a9 not found: ID does not exist" Oct 30 06:47:33.999518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.999416 5001 scope.go:122] "RemoveContainer" containerID="5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f" Oct 30 06:47:33.999780 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:33.999748 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f\": container with ID starting with 5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f not found: ID does not exist" containerID="5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f" Oct 30 06:47:33.999891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:33.999790 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f"} err="failed to get container status \"5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f\": rpc error: code = NotFound desc = could not find container \"5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f\": container with ID starting with 5cddf2ec043e7c7d2b30ee398026e23f4877c7956f9e462d8a43a6bb15528b3f not found: ID does not exist" Oct 30 06:47:34.001362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:34.001340 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn"] Oct 30 06:47:34.013272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:34.013247 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cn5xn"] Oct 30 06:47:34.052908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:34.052845 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg"] Oct 30 06:47:34.061139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:34.061104 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-fhhtg"] Oct 30 06:47:34.079667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:34.079631 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4"] Oct 30 06:47:34.122276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:34.122235 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" path="/var/lib/kubelet/pods/b2bf08e0-cac0-43ed-847a-3df6d9003e3c/volumes" Oct 30 06:47:34.123226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:34.123051 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" path="/var/lib/kubelet/pods/f5fc25a2-998b-4a99-8ab2-ac5d4b11f415/volumes" Oct 30 06:47:34.760040 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:34.759997 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p"] Oct 30 06:47:34.760562 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:34.760221 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" podUID="7c3862d1-263a-42e6-a01f-90b4ee0629dc" containerName="sim" containerID="cri-o://56403ee78f639d3062ea035921811d9436ee3d7ed6fd141a37894a230091f714" gracePeriod=30 Oct 30 06:47:34.944543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:34.944513 5001 generic.go:356] "Generic (PLEG): container finished" podID="7c3862d1-263a-42e6-a01f-90b4ee0629dc" containerID="56403ee78f639d3062ea035921811d9436ee3d7ed6fd141a37894a230091f714" exitCode=0 Oct 30 06:47:34.944543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:34.944552 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" event={"ID":"7c3862d1-263a-42e6-a01f-90b4ee0629dc","Type":"ContainerDied","Data":"56403ee78f639d3062ea035921811d9436ee3d7ed6fd141a37894a230091f714"} Oct 30 06:47:35.012511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.012429 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks"] Oct 30 06:47:35.104166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.104142 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:47:35.211145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.211109 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3862d1-263a-42e6-a01f-90b4ee0629dc-tls\" (UniqueName: \"kubernetes.io/secret/7c3862d1-263a-42e6-a01f-90b4ee0629dc-tls\") pod \"7c3862d1-263a-42e6-a01f-90b4ee0629dc\" (UID: \"7c3862d1-263a-42e6-a01f-90b4ee0629dc\") " Oct 30 06:47:35.211350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.211266 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3862d1-263a-42e6-a01f-90b4ee0629dc-kube-api-access-57dsw\" (UniqueName: \"kubernetes.io/projected/7c3862d1-263a-42e6-a01f-90b4ee0629dc-kube-api-access-57dsw\") pod \"7c3862d1-263a-42e6-a01f-90b4ee0629dc\" (UID: \"7c3862d1-263a-42e6-a01f-90b4ee0629dc\") " Oct 30 06:47:35.213519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.213488 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3862d1-263a-42e6-a01f-90b4ee0629dc-tls" pod "7c3862d1-263a-42e6-a01f-90b4ee0629dc" (UID: "7c3862d1-263a-42e6-a01f-90b4ee0629dc"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:47:35.213669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.213625 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3862d1-263a-42e6-a01f-90b4ee0629dc-kube-api-access-57dsw" pod "7c3862d1-263a-42e6-a01f-90b4ee0629dc" (UID: "7c3862d1-263a-42e6-a01f-90b4ee0629dc"). InnerVolumeSpecName "kube-api-access-57dsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:35.334912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.330700 5001 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7c3862d1-263a-42e6-a01f-90b4ee0629dc-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:35.334912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.330745 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57dsw\" (UniqueName: \"kubernetes.io/projected/7c3862d1-263a-42e6-a01f-90b4ee0629dc-kube-api-access-57dsw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:35.824051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.824021 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:47:35.824522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.824128 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:47:35.964633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.964605 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" Oct 30 06:47:35.964845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.964596 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p" event={"ID":"7c3862d1-263a-42e6-a01f-90b4ee0629dc","Type":"ContainerDied","Data":"324caed7686c85e03b466d0d4559e67adf57edad83d9c4af9cb2db66421abc48"} Oct 30 06:47:35.964845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:35.964746 5001 scope.go:122] "RemoveContainer" containerID="56403ee78f639d3062ea035921811d9436ee3d7ed6fd141a37894a230091f714" Oct 30 06:47:36.026507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:36.023240 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p"] Oct 30 06:47:36.036178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:36.036143 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-f08338/e2e-usage-sim-b3afd6-9b9d49b5b-m2x8p"] Oct 30 06:47:36.119258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:36.119227 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3862d1-263a-42e6-a01f-90b4ee0629dc" path="/var/lib/kubelet/pods/7c3862d1-263a-42e6-a01f-90b4ee0629dc/volumes" Oct 30 06:47:38.270611 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:38.270574 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:47:39.080410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.080355 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" podUID="2bd4206d-f00c-48ad-97b3-e0feedf8e7ba" containerName="payload-processing" containerID="cri-o://00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d" gracePeriod=25 Oct 30 06:47:39.387478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.387447 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:47:39.507287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.507254 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-plugins-config-volume\") pod \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " Oct 30 06:47:39.507502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.507370 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-provider-credentials\") pod \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " Oct 30 06:47:39.507502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.507405 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-kube-api-access-7ncfr\" (UniqueName: \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-kube-api-access-7ncfr\") pod \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " Oct 30 06:47:39.507712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.507692 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-routing-overlay-volume\") pod \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\" (UID: \"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba\") " Oct 30 06:47:39.507779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.507716 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-plugins-config-volume" pod "2bd4206d-f00c-48ad-97b3-e0feedf8e7ba" (UID: "2bd4206d-f00c-48ad-97b3-e0feedf8e7ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:39.508067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.508041 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-routing-overlay-volume" pod "2bd4206d-f00c-48ad-97b3-e0feedf8e7ba" (UID: "2bd4206d-f00c-48ad-97b3-e0feedf8e7ba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:39.508550 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.508533 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:39.508609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.508554 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:39.509879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.509836 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-provider-credentials" pod "2bd4206d-f00c-48ad-97b3-e0feedf8e7ba" (UID: "2bd4206d-f00c-48ad-97b3-e0feedf8e7ba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:39.509981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.509920 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-kube-api-access-7ncfr" pod "2bd4206d-f00c-48ad-97b3-e0feedf8e7ba" (UID: "2bd4206d-f00c-48ad-97b3-e0feedf8e7ba"). InnerVolumeSpecName "kube-api-access-7ncfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:39.609657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.609625 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:39.609657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.609658 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ncfr\" (UniqueName: \"kubernetes.io/projected/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba-kube-api-access-7ncfr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:39.904496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:39.904403 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:47:40.012743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.012699 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" podUID="6d41c70e-ac2f-47c9-9c71-3adf2505b04b" containerName="payload-processing" containerID="cri-o://fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0" gracePeriod=25 Oct 30 06:47:40.015354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.015115 5001 generic.go:356] "Generic (PLEG): container finished" podID="2bd4206d-f00c-48ad-97b3-e0feedf8e7ba" containerID="00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d" exitCode=0 Oct 30 06:47:40.015354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.015147 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" event={"ID":"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba","Type":"ContainerDied","Data":"00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d"} Oct 30 06:47:40.015354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.015341 5001 scope.go:122] "RemoveContainer" containerID="00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d" Oct 30 06:47:40.015643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.015395 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" Oct 30 06:47:40.015643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.015326 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4" event={"ID":"2bd4206d-f00c-48ad-97b3-e0feedf8e7ba","Type":"ContainerDied","Data":"a8176ace67ada5ae43f0cab0cf15e5aa172e5e360c019e7e6b00edc971de9577"} Oct 30 06:47:40.041230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.041197 5001 scope.go:122] "RemoveContainer" containerID="00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d" Oct 30 06:47:40.041511 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:40.041486 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d\": container with ID starting with 00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d not found: ID does not exist" containerID="00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d" Oct 30 06:47:40.041568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.041522 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d"} err="failed to get container status \"00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d\": rpc error: code = NotFound desc = could not find container \"00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d\": container with ID starting with 00d4770771234eb1c85e557c39f784f6d4be9b5d27db991b2dc710d4d5171e3d not found: ID does not exist" Oct 30 06:47:40.076709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.076653 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4"] Oct 30 06:47:40.094325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.086262 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6768wn4x4"] Oct 30 06:47:40.123097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.123027 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd4206d-f00c-48ad-97b3-e0feedf8e7ba" path="/var/lib/kubelet/pods/2bd4206d-f00c-48ad-97b3-e0feedf8e7ba/volumes" Oct 30 06:47:40.341601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.341571 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:47:40.422436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.422396 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-routing-overlay-volume\") pod \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " Oct 30 06:47:40.422918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.422495 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-provider-credentials\") pod \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " Oct 30 06:47:40.422918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.422622 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-kube-api-access-jx88d\" (UniqueName: \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-kube-api-access-jx88d\") pod \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " Oct 30 06:47:40.422918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.422659 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-plugins-config-volume\") pod \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\" (UID: \"6d41c70e-ac2f-47c9-9c71-3adf2505b04b\") " Oct 30 06:47:40.422918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.422835 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-routing-overlay-volume" pod "6d41c70e-ac2f-47c9-9c71-3adf2505b04b" (UID: "6d41c70e-ac2f-47c9-9c71-3adf2505b04b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:40.423154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.423103 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-plugins-config-volume" pod "6d41c70e-ac2f-47c9-9c71-3adf2505b04b" (UID: "6d41c70e-ac2f-47c9-9c71-3adf2505b04b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:40.423490 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.423462 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:40.423571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.423490 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:40.426742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.426711 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-kube-api-access-jx88d" pod "6d41c70e-ac2f-47c9-9c71-3adf2505b04b" (UID: "6d41c70e-ac2f-47c9-9c71-3adf2505b04b"). InnerVolumeSpecName "kube-api-access-jx88d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:40.427671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.427649 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-provider-credentials" pod "6d41c70e-ac2f-47c9-9c71-3adf2505b04b" (UID: "6d41c70e-ac2f-47c9-9c71-3adf2505b04b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:40.526025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.525984 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:40.526025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:40.526025 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx88d\" (UniqueName: \"kubernetes.io/projected/6d41c70e-ac2f-47c9-9c71-3adf2505b04b-kube-api-access-jx88d\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:47:41.029400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:41.029368 5001 generic.go:356] "Generic (PLEG): container finished" podID="6d41c70e-ac2f-47c9-9c71-3adf2505b04b" containerID="fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0" exitCode=0 Oct 30 06:47:41.029595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:41.029428 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" Oct 30 06:47:41.029595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:41.029458 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" event={"ID":"6d41c70e-ac2f-47c9-9c71-3adf2505b04b","Type":"ContainerDied","Data":"fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0"} Oct 30 06:47:41.029595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:41.029519 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks" event={"ID":"6d41c70e-ac2f-47c9-9c71-3adf2505b04b","Type":"ContainerDied","Data":"360e1f2615e36511f914feb1d37df300c2998f872e2682aae10a4ab67df96633"} Oct 30 06:47:41.029595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:41.029543 5001 scope.go:122] "RemoveContainer" containerID="fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0" Oct 30 06:47:41.043759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:41.043743 5001 scope.go:122] "RemoveContainer" containerID="fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0" Oct 30 06:47:41.044046 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:47:41.044025 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0\": container with ID starting with fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0 not found: ID does not exist" containerID="fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0" Oct 30 06:47:41.044099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:41.044054 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0"} err="failed to get container status \"fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0\": rpc error: code = NotFound desc = could not find container \"fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0\": container with ID starting with fa8cc3a895709057b3571958db3e17ccab378ae614d4fe3c398f2ed5f4ab52e0 not found: ID does not exist" Oct 30 06:47:41.087849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:41.087817 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks"] Oct 30 06:47:41.092417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:41.092389 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6986rh5ks"] Oct 30 06:47:42.122636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:42.122595 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d41c70e-ac2f-47c9-9c71-3adf2505b04b" path="/var/lib/kubelet/pods/6d41c70e-ac2f-47c9-9c71-3adf2505b04b/volumes" Oct 30 06:47:42.947421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:42.947374 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv"] Oct 30 06:47:42.981333 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:42.948889 5001 cpu_manager.go:414] "containerMap: removing container" podUID="84b8e92e-ca86-4a1d-ada3-f99d7f5de94d" containerName="revoke-keys" Oct 30 06:47:43.071074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.071039 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b8e92e-ca86-4a1d-ada3-f99d7f5de94d" containerName="revoke-keys" Oct 30 06:47:43.071400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.071378 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" Oct 30 06:47:43.090522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.090482 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" Oct 30 06:47:43.090770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.090753 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2bd4206d-f00c-48ad-97b3-e0feedf8e7ba" containerName="payload-processing" Oct 30 06:47:43.094381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.094344 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd4206d-f00c-48ad-97b3-e0feedf8e7ba" containerName="payload-processing" Oct 30 06:47:43.102397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.102364 5001 cpu_manager.go:414] "containerMap: removing container" podUID="121c8afc-303a-449b-81dc-8c3e795b598d" containerName="payload-processing" Oct 30 06:47:43.102597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.102580 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121c8afc-303a-449b-81dc-8c3e795b598d" containerName="payload-processing" Oct 30 06:47:43.102762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.102751 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7c3862d1-263a-42e6-a01f-90b4ee0629dc" containerName="sim" Oct 30 06:47:43.102861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.102849 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3862d1-263a-42e6-a01f-90b4ee0629dc" containerName="sim" Oct 30 06:47:43.102953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.102942 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" containerName="istio-proxy" Oct 30 06:47:43.103027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.103018 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" containerName="istio-proxy" Oct 30 06:47:43.119345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.103116 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6d41c70e-ac2f-47c9-9c71-3adf2505b04b" containerName="payload-processing" Oct 30 06:47:43.119580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.119555 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d41c70e-ac2f-47c9-9c71-3adf2505b04b" containerName="payload-processing" Oct 30 06:47:43.119724 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.119713 5001 cpu_manager.go:414] "containerMap: removing container" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerName="istio-proxy" Oct 30 06:47:43.119807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.119796 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerName="istio-proxy" Oct 30 06:47:43.120175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.120158 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" Oct 30 06:47:43.120300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.120288 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3862d1-263a-42e6-a01f-90b4ee0629dc" containerName="sim" Oct 30 06:47:43.120441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.120429 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fc25a2-998b-4a99-8ab2-ac5d4b11f415" containerName="istio-proxy" Oct 30 06:47:43.120528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.120518 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d41c70e-ac2f-47c9-9c71-3adf2505b04b" containerName="payload-processing" Oct 30 06:47:43.120607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.120598 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a380d4-e404-41ed-9e49-1af4f8b822d8" containerName="istio-proxy" Oct 30 06:47:43.120694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.120684 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b8e92e-ca86-4a1d-ada3-f99d7f5de94d" containerName="revoke-keys" Oct 30 06:47:43.120794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.120783 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd4206d-f00c-48ad-97b3-e0feedf8e7ba" containerName="payload-processing" Oct 30 06:47:43.120877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.120867 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="121c8afc-303a-449b-81dc-8c3e795b598d" containerName="payload-processing" Oct 30 06:47:43.430331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.405452 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv"] Oct 30 06:47:43.430848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.406636 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.462645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.462612 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b22a13-openshift-default-dockercfg-scj6c\"" Oct 30 06:47:43.534332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.534041 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bbf981-0119-4ab2-9a8d-51e21306a98d-istiod-ca-cert\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.534332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.534104 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-certs\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.534332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.534140 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.534332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.534236 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh6k4\" (UniqueName: \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-kube-api-access-dh6k4\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.534332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.534274 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-data\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.534796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.534386 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-token\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.534796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.534428 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-credential-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.534796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.534496 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-podinfo\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.534796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.534553 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-envoy\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.637334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.637282 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6k4\" (UniqueName: \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-kube-api-access-dh6k4\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.646996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.646956 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-data\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.648129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.648099 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-token\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.648438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.648412 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-credential-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.648663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.648644 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-podinfo\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.648841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.648824 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-envoy\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.649146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.649126 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bbf981-0119-4ab2-9a8d-51e21306a98d-istiod-ca-cert\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.684535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.676479 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-certs\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.684535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.676530 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.684535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.666775 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bbf981-0119-4ab2-9a8d-51e21306a98d-istiod-ca-cert\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.684535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.649557 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-credential-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.684535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.677101 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-certs\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.684535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.647961 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-data\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.691566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.691531 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.727332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.723709 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-podinfo\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.738809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.724056 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh6k4\" (UniqueName: \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-kube-api-access-dh6k4\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.739059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.738726 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-envoy\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.770518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.747939 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-token\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:43.829527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:43.828153 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:44.617090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:44.617055 5001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4fh4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:47:44.617835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:44.617784 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:47:44.934243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:44.934175 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:47:44.937555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:44.934492 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:47:44.937555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:44.934614 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:47:45.086328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:45.082214 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv"] Oct 30 06:47:45.370675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:45.370636 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" event={"ID":"06bbf981-0119-4ab2-9a8d-51e21306a98d","Type":"ContainerStarted","Data":"f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df"} Oct 30 06:47:45.371371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:45.370704 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" event={"ID":"06bbf981-0119-4ab2-9a8d-51e21306a98d","Type":"ContainerStarted","Data":"a76f710c353376bb86787186530fdc2775dfa473c74d399de4821b44cf2d47e0"} Oct 30 06:47:45.427987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:45.427934 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" podStartSLOduration=3.427916468 podStartE2EDuration="3.427916468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:45.425609661 +0000 UTC m=+3279.421566675" watchObservedRunningTime="2026-10-30 06:47:45.427916468 +0000 UTC m=+3279.423873478" Oct 30 06:47:45.836357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:45.831404 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:46.835281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:46.835179 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:47.390369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:47.390330 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:47.391448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:47.391428 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:47:49.749734 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.749687 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4"] Oct 30 06:47:49.750634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.750615 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" Oct 30 06:47:49.750710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.750636 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" Oct 30 06:47:49.750760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.750754 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bf08e0-cac0-43ed-847a-3df6d9003e3c" containerName="istio-proxy" Oct 30 06:47:49.755037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.755016 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:47:49.775539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.775509 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4"] Oct 30 06:47:49.781045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.781019 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b22a13-dockercfg-ks5cm\"" Oct 30 06:47:49.781266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.781072 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b22a13\"" Oct 30 06:47:49.815496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.815460 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps"] Oct 30 06:47:49.820239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.820217 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:47:49.837170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.837138 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps"] Oct 30 06:47:49.952253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.952208 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7d8325-1c73-43af-bea6-5f14fe875bdf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4\" (UID: \"ad7d8325-1c73-43af-bea6-5f14fe875bdf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:47:49.952700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.952675 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k52wq\" (UniqueName: \"kubernetes.io/projected/f616e0d8-f098-4e17-8dfc-ce9076813eb0-kube-api-access-k52wq\") pod \"payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps\" (UID: \"f616e0d8-f098-4e17-8dfc-ce9076813eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:47:49.952902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.952885 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f616e0d8-f098-4e17-8dfc-ce9076813eb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps\" (UID: \"f616e0d8-f098-4e17-8dfc-ce9076813eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:47:49.953065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:49.953049 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ld7s\" (UniqueName: \"kubernetes.io/projected/ad7d8325-1c73-43af-bea6-5f14fe875bdf-kube-api-access-5ld7s\") pod \"payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4\" (UID: \"ad7d8325-1c73-43af-bea6-5f14fe875bdf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:47:50.054633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.054547 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ld7s\" (UniqueName: \"kubernetes.io/projected/ad7d8325-1c73-43af-bea6-5f14fe875bdf-kube-api-access-5ld7s\") pod \"payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4\" (UID: \"ad7d8325-1c73-43af-bea6-5f14fe875bdf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:47:50.076113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.076077 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7d8325-1c73-43af-bea6-5f14fe875bdf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4\" (UID: \"ad7d8325-1c73-43af-bea6-5f14fe875bdf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:47:50.082529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.082497 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52wq\" (UniqueName: \"kubernetes.io/projected/f616e0d8-f098-4e17-8dfc-ce9076813eb0-kube-api-access-k52wq\") pod \"payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps\" (UID: \"f616e0d8-f098-4e17-8dfc-ce9076813eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:47:50.083849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.083824 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f616e0d8-f098-4e17-8dfc-ce9076813eb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps\" (UID: \"f616e0d8-f098-4e17-8dfc-ce9076813eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:47:50.085365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.077230 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7d8325-1c73-43af-bea6-5f14fe875bdf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4\" (UID: \"ad7d8325-1c73-43af-bea6-5f14fe875bdf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:47:50.085556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.085164 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f616e0d8-f098-4e17-8dfc-ce9076813eb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps\" (UID: \"f616e0d8-f098-4e17-8dfc-ce9076813eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:47:50.093731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.093581 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ld7s\" (UniqueName: \"kubernetes.io/projected/ad7d8325-1c73-43af-bea6-5f14fe875bdf-kube-api-access-5ld7s\") pod \"payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4\" (UID: \"ad7d8325-1c73-43af-bea6-5f14fe875bdf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:47:50.150327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.147085 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52wq\" (UniqueName: \"kubernetes.io/projected/f616e0d8-f098-4e17-8dfc-ce9076813eb0-kube-api-access-k52wq\") pod \"payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps\" (UID: \"f616e0d8-f098-4e17-8dfc-ce9076813eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:47:50.378036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.378000 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:47:50.437402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.437281 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:47:50.742084 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:47:50.742057 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7d8325_1c73_43af_bea6_5f14fe875bdf.slice/crio-0cf86613f988da247b3fda614d2f2140e7349706a049d743561d10cb08846ce8 WatchSource:0}: Error finding container 0cf86613f988da247b3fda614d2f2140e7349706a049d743561d10cb08846ce8: Status 404 returned error can't find the container with id 0cf86613f988da247b3fda614d2f2140e7349706a049d743561d10cb08846ce8 Oct 30 06:47:50.742267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.742245 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4"] Oct 30 06:47:50.804016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:50.803987 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps"] Oct 30 06:47:50.805608 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:47:50.805583 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf616e0d8_f098_4e17_8dfc_ce9076813eb0.slice/crio-4b1ad848d3dfc3522cc0080b251ec0f32f74bd61424e14b28df42721d71c261b WatchSource:0}: Error finding container 4b1ad848d3dfc3522cc0080b251ec0f32f74bd61424e14b28df42721d71c261b: Status 404 returned error can't find the container with id 4b1ad848d3dfc3522cc0080b251ec0f32f74bd61424e14b28df42721d71c261b Oct 30 06:47:51.431466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:51.431428 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" event={"ID":"f616e0d8-f098-4e17-8dfc-ce9076813eb0","Type":"ContainerStarted","Data":"11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe"} Oct 30 06:47:51.431680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:51.431487 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" event={"ID":"f616e0d8-f098-4e17-8dfc-ce9076813eb0","Type":"ContainerStarted","Data":"4b1ad848d3dfc3522cc0080b251ec0f32f74bd61424e14b28df42721d71c261b"} Oct 30 06:47:51.431680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:51.431523 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:47:51.433458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:51.433430 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" event={"ID":"ad7d8325-1c73-43af-bea6-5f14fe875bdf","Type":"ContainerStarted","Data":"5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99"} Oct 30 06:47:51.433986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:51.433472 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" event={"ID":"ad7d8325-1c73-43af-bea6-5f14fe875bdf","Type":"ContainerStarted","Data":"0cf86613f988da247b3fda614d2f2140e7349706a049d743561d10cb08846ce8"} Oct 30 06:47:51.433986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:51.433491 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:47:51.480687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:51.480630 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" podStartSLOduration=2.480612361 podStartE2EDuration="2.480612361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:51.473556085 +0000 UTC m=+3285.469513098" watchObservedRunningTime="2026-10-30 06:47:51.480612361 +0000 UTC m=+3285.476569372" Oct 30 06:47:51.515502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:47:51.515453 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" podStartSLOduration=2.515439434 podStartE2EDuration="2.515439434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:51.514721768 +0000 UTC m=+3285.510678779" watchObservedRunningTime="2026-10-30 06:47:51.515439434 +0000 UTC m=+3285.511396446" Oct 30 06:48:02.466452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:02.466417 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:48:02.468535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:02.468509 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:48:06.067095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.067058 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9"] Oct 30 06:48:06.073231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.073210 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:06.091936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.091907 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9"] Oct 30 06:48:06.100364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.098701 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:48:06.100364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.098769 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:48:06.100364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.098917 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:48:06.102566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.102530 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:48:06.109269 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.108245 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1af0fc-9lph9\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:06.109269 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.108348 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1af0fc-9lph9\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:06.109269 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.108463 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsvjg\" (UniqueName: \"kubernetes.io/projected/3c8989eb-ab37-431b-b217-bed1486e939d-kube-api-access-qsvjg\") pod \"maas-api-revoke-sub-ff1af0fc-9lph9\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:06.144104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.144065 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp"] Oct 30 06:48:06.149367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.149341 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp"] Oct 30 06:48:06.149541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.149385 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:06.209639 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.209590 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1af0fc-9lph9\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:06.209856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.209671 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1af0fc-9lph9\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:06.209856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.209790 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsvjg\" (UniqueName: \"kubernetes.io/projected/3c8989eb-ab37-431b-b217-bed1486e939d-kube-api-access-qsvjg\") pod \"maas-api-revoke-sub-ff1af0fc-9lph9\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:06.209996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.209868 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad1f1ff-jrnwp\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:06.210087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.210043 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad1f1ff-jrnwp\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:06.210192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.210174 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd5s6\" (UniqueName: \"kubernetes.io/projected/99bdccb2-468d-4cfe-9687-f37ad6391739-kube-api-access-bd5s6\") pod \"maas-api-revoke-sub-4ad1f1ff-jrnwp\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:06.210447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.210424 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1af0fc-9lph9\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:06.224134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.224105 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:48:06.233028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.233002 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1af0fc-9lph9\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:06.245791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.245736 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:48:06.255118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.255089 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsvjg\" (UniqueName: \"kubernetes.io/projected/3c8989eb-ab37-431b-b217-bed1486e939d-kube-api-access-qsvjg\") pod \"maas-api-revoke-sub-ff1af0fc-9lph9\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:06.311629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.311593 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad1f1ff-jrnwp\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:06.312109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.311963 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad1f1ff-jrnwp\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:06.312109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.312091 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd5s6\" (UniqueName: \"kubernetes.io/projected/99bdccb2-468d-4cfe-9687-f37ad6391739-kube-api-access-bd5s6\") pod \"maas-api-revoke-sub-4ad1f1ff-jrnwp\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:06.313185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.313128 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad1f1ff-jrnwp\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:06.315692 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.315669 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad1f1ff-jrnwp\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:06.336743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.336677 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd5s6\" (UniqueName: \"kubernetes.io/projected/99bdccb2-468d-4cfe-9687-f37ad6391739-kube-api-access-bd5s6\") pod \"maas-api-revoke-sub-4ad1f1ff-jrnwp\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:06.409053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.409017 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:48:06.413966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.413933 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:06.469947 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.469904 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:06.755326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.755266 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9"] Oct 30 06:48:06.760603 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:48:06.760578 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8989eb_ab37_431b_b217_bed1486e939d.slice/crio-344c9f604f2ac8ac30bdd6aeeaf54402228bfcb2fdf067a44c7c8523ed5f185c WatchSource:0}: Error finding container 344c9f604f2ac8ac30bdd6aeeaf54402228bfcb2fdf067a44c7c8523ed5f185c: Status 404 returned error can't find the container with id 344c9f604f2ac8ac30bdd6aeeaf54402228bfcb2fdf067a44c7c8523ed5f185c Oct 30 06:48:06.811094 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:48:06.811060 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bdccb2_468d_4cfe_9687_f37ad6391739.slice/crio-45254466ba6e2d17d284b3c98b21c81a75863c36840781ef9191c099c26cddb6 WatchSource:0}: Error finding container 45254466ba6e2d17d284b3c98b21c81a75863c36840781ef9191c099c26cddb6: Status 404 returned error can't find the container with id 45254466ba6e2d17d284b3c98b21c81a75863c36840781ef9191c099c26cddb6 Oct 30 06:48:06.811205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:06.811135 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp"] Oct 30 06:48:07.058811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:07.058719 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:48:07.073879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:07.073855 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:48:07.748431 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:07.748333 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" event={"ID":"3c8989eb-ab37-431b-b217-bed1486e939d","Type":"ContainerStarted","Data":"d63be31702b9f93109a5dc6d4dee266627f616963b427ed9c356b34a951cf7af"} Oct 30 06:48:07.748591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:07.748430 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" event={"ID":"3c8989eb-ab37-431b-b217-bed1486e939d","Type":"ContainerStarted","Data":"344c9f604f2ac8ac30bdd6aeeaf54402228bfcb2fdf067a44c7c8523ed5f185c"} Oct 30 06:48:07.750253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:07.750227 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" event={"ID":"99bdccb2-468d-4cfe-9687-f37ad6391739","Type":"ContainerStarted","Data":"01d82343070266aac2fb4d59b57ee49ac332d3b684bf79df38a1d80e1af49c74"} Oct 30 06:48:07.750388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:07.750275 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" event={"ID":"99bdccb2-468d-4cfe-9687-f37ad6391739","Type":"ContainerStarted","Data":"45254466ba6e2d17d284b3c98b21c81a75863c36840781ef9191c099c26cddb6"} Oct 30 06:48:07.806636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:07.806583 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" podStartSLOduration=1.806569386 podStartE2EDuration="1.806569386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:07.789670381 +0000 UTC m=+3301.785627390" watchObservedRunningTime="2026-10-30 06:48:07.806569386 +0000 UTC m=+3301.802526396" Oct 30 06:48:07.835875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:07.835820 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" podStartSLOduration=1.8358054990000001 podStartE2EDuration="1.835805499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:07.830526266 +0000 UTC m=+3301.826483277" watchObservedRunningTime="2026-10-30 06:48:07.835805499 +0000 UTC m=+3301.831762543" Oct 30 06:48:08.271704 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:08.271659 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:48:08.764148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:08.764111 5001 generic.go:356] "Generic (PLEG): container finished" podID="3c8989eb-ab37-431b-b217-bed1486e939d" containerID="d63be31702b9f93109a5dc6d4dee266627f616963b427ed9c356b34a951cf7af" exitCode=0 Oct 30 06:48:08.764372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:08.764194 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" event={"ID":"3c8989eb-ab37-431b-b217-bed1486e939d","Type":"ContainerDied","Data":"d63be31702b9f93109a5dc6d4dee266627f616963b427ed9c356b34a951cf7af"} Oct 30 06:48:08.771124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:08.771088 5001 generic.go:356] "Generic (PLEG): container finished" podID="99bdccb2-468d-4cfe-9687-f37ad6391739" containerID="01d82343070266aac2fb4d59b57ee49ac332d3b684bf79df38a1d80e1af49c74" exitCode=0 Oct 30 06:48:08.771288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:08.771173 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" event={"ID":"99bdccb2-468d-4cfe-9687-f37ad6391739","Type":"ContainerDied","Data":"01d82343070266aac2fb4d59b57ee49ac332d3b684bf79df38a1d80e1af49c74"} Oct 30 06:48:10.091123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.091096 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:10.095458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.095434 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:10.199209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.199171 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-xks-ca\") pod \"99bdccb2-468d-4cfe-9687-f37ad6391739\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " Oct 30 06:48:10.199407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.199263 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8989eb-ab37-431b-b217-bed1486e939d-kube-api-access-qsvjg\" (UniqueName: \"kubernetes.io/projected/3c8989eb-ab37-431b-b217-bed1486e939d-kube-api-access-qsvjg\") pod \"3c8989eb-ab37-431b-b217-bed1486e939d\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " Oct 30 06:48:10.199407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.199332 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-openshift-ca\") pod \"99bdccb2-468d-4cfe-9687-f37ad6391739\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " Oct 30 06:48:10.199407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.199370 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-xks-ca\") pod \"3c8989eb-ab37-431b-b217-bed1486e939d\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " Oct 30 06:48:10.199617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.199429 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-openshift-ca\") pod \"3c8989eb-ab37-431b-b217-bed1486e939d\" (UID: \"3c8989eb-ab37-431b-b217-bed1486e939d\") " Oct 30 06:48:10.199617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.199466 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99bdccb2-468d-4cfe-9687-f37ad6391739-kube-api-access-bd5s6\" (UniqueName: \"kubernetes.io/projected/99bdccb2-468d-4cfe-9687-f37ad6391739-kube-api-access-bd5s6\") pod \"99bdccb2-468d-4cfe-9687-f37ad6391739\" (UID: \"99bdccb2-468d-4cfe-9687-f37ad6391739\") " Oct 30 06:48:10.200070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.200038 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-openshift-ca" pod "3c8989eb-ab37-431b-b217-bed1486e939d" (UID: "3c8989eb-ab37-431b-b217-bed1486e939d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:48:10.200070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.200051 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-openshift-ca" pod "99bdccb2-468d-4cfe-9687-f37ad6391739" (UID: "99bdccb2-468d-4cfe-9687-f37ad6391739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:48:10.200847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.200823 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:48:10.200847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.200849 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:48:10.202257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.202228 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-xks-ca" pod "99bdccb2-468d-4cfe-9687-f37ad6391739" (UID: "99bdccb2-468d-4cfe-9687-f37ad6391739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:48:10.202909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.202883 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8989eb-ab37-431b-b217-bed1486e939d-kube-api-access-qsvjg" pod "3c8989eb-ab37-431b-b217-bed1486e939d" (UID: "3c8989eb-ab37-431b-b217-bed1486e939d"). InnerVolumeSpecName "kube-api-access-qsvjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:48:10.202982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.202918 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bdccb2-468d-4cfe-9687-f37ad6391739-kube-api-access-bd5s6" pod "99bdccb2-468d-4cfe-9687-f37ad6391739" (UID: "99bdccb2-468d-4cfe-9687-f37ad6391739"). InnerVolumeSpecName "kube-api-access-bd5s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:48:10.203771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.203752 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-xks-ca" pod "3c8989eb-ab37-431b-b217-bed1486e939d" (UID: "3c8989eb-ab37-431b-b217-bed1486e939d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:48:10.302990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.302902 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsvjg\" (UniqueName: \"kubernetes.io/projected/3c8989eb-ab37-431b-b217-bed1486e939d-kube-api-access-qsvjg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:48:10.302990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.302934 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8989eb-ab37-431b-b217-bed1486e939d-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:48:10.302990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.302949 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd5s6\" (UniqueName: \"kubernetes.io/projected/99bdccb2-468d-4cfe-9687-f37ad6391739-kube-api-access-bd5s6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:48:10.302990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.302962 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bdccb2-468d-4cfe-9687-f37ad6391739-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:48:10.795778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.795745 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" Oct 30 06:48:10.796011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.795788 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp" event={"ID":"99bdccb2-468d-4cfe-9687-f37ad6391739","Type":"ContainerDied","Data":"45254466ba6e2d17d284b3c98b21c81a75863c36840781ef9191c099c26cddb6"} Oct 30 06:48:10.796011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.795847 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45254466ba6e2d17d284b3c98b21c81a75863c36840781ef9191c099c26cddb6" Oct 30 06:48:10.797587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.797553 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" Oct 30 06:48:10.797587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.797563 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9" event={"ID":"3c8989eb-ab37-431b-b217-bed1486e939d","Type":"ContainerDied","Data":"344c9f604f2ac8ac30bdd6aeeaf54402228bfcb2fdf067a44c7c8523ed5f185c"} Oct 30 06:48:10.797822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:10.797609 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="344c9f604f2ac8ac30bdd6aeeaf54402228bfcb2fdf067a44c7c8523ed5f185c" Oct 30 06:48:11.465898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:11.465859 5001 scope.go:122] "RemoveContainer" containerID="baa350eff7c141d661e6c4998685280505910a7c76b0792472b28a9dec179e71" Oct 30 06:48:11.479151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:11.479126 5001 scope.go:122] "RemoveContainer" containerID="d5477d5d47f535fc0a33fabbf27d88a0a7a888d22f997abfa3356079fd733511" Oct 30 06:48:11.490201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:11.490179 5001 scope.go:122] "RemoveContainer" containerID="254162814ccc56ae7fef298568305be7c98276912e2082f1f793133126f88b85" Oct 30 06:48:11.502242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:11.502217 5001 scope.go:122] "RemoveContainer" containerID="cf1e468a0806872eb07ddd7bb49cb2f3bf3fa2df5281660072cf387a0556b076" Oct 30 06:48:11.512373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:11.512348 5001 scope.go:122] "RemoveContainer" containerID="66228458c3974ee2b7b7150e7b297c0a9bc42d998c824842d28c638fd0bd04a9" Oct 30 06:48:17.184370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:17.180715 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:48:23.141385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.139934 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq"] Oct 30 06:48:23.141385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.141068 5001 cpu_manager.go:414] "containerMap: removing container" podUID="99bdccb2-468d-4cfe-9687-f37ad6391739" containerName="revoke-keys" Oct 30 06:48:23.141385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.141089 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bdccb2-468d-4cfe-9687-f37ad6391739" containerName="revoke-keys" Oct 30 06:48:23.141385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.141134 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3c8989eb-ab37-431b-b217-bed1486e939d" containerName="revoke-keys" Oct 30 06:48:23.141385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.141142 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8989eb-ab37-431b-b217-bed1486e939d" containerName="revoke-keys" Oct 30 06:48:23.141385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.141389 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8989eb-ab37-431b-b217-bed1486e939d" containerName="revoke-keys" Oct 30 06:48:23.142403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.141422 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bdccb2-468d-4cfe-9687-f37ad6391739" containerName="revoke-keys" Oct 30 06:48:23.148106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.148083 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq"] Oct 30 06:48:23.148245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.148166 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.193235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.193195 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"openshift-service-ca.crt\"" Oct 30 06:48:23.193436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.193203 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"kube-root-ca.crt\"" Oct 30 06:48:23.193436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.193264 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"payload-processing-external-model-e2e--d83828a5-dockercfg-jvkn2\"" Oct 30 06:48:23.193436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.193203 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2d2923\"" Oct 30 06:48:23.193436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.193203 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"routing-overlay\"" Oct 30 06:48:23.193436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.193203 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"agc-cred-cdd339524c27\"" Oct 30 06:48:23.267331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.266932 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.267331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.266998 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll9pj\" (UniqueName: \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-kube-api-access-ll9pj\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.267331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.267168 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.267331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.267276 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.373957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.373902 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.374414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.374387 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.374902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.374881 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.375066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.375050 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9pj\" (UniqueName: \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-kube-api-access-ll9pj\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.399677 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.399596 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.411388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.400201 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.432330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.427238 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.434718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.434669 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll9pj\" (UniqueName: \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-kube-api-access-ll9pj\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.502916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.502596 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:23.932521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:23.932489 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq"] Oct 30 06:48:23.932649 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:48:23.932519 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5969015_c270_4671_8af3_d5a565fa48a3.slice/crio-730393e3899f9865bbe802d0ba5560aa0eeb75adbbe843d0f82b10440fa616ec WatchSource:0}: Error finding container 730393e3899f9865bbe802d0ba5560aa0eeb75adbbe843d0f82b10440fa616ec: Status 404 returned error can't find the container with id 730393e3899f9865bbe802d0ba5560aa0eeb75adbbe843d0f82b10440fa616ec Oct 30 06:48:24.190010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.189879 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" event={"ID":"f5969015-c270-4671-8af3-d5a565fa48a3","Type":"ContainerStarted","Data":"730393e3899f9865bbe802d0ba5560aa0eeb75adbbe843d0f82b10440fa616ec"} Oct 30 06:48:24.190010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.189952 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn"] Oct 30 06:48:24.204710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.203789 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn"] Oct 30 06:48:24.204710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.203945 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.220180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.220155 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"payload-processing-external-model-e2e--b62900db-dockercfg-zgffj\"" Oct 30 06:48:24.220264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.220202 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"agc-cred-68ded8fa1796\"" Oct 30 06:48:24.220765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.220743 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"openshift-service-ca.crt\"" Oct 30 06:48:24.220837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.220816 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"kube-root-ca.crt\"" Oct 30 06:48:24.220876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.220855 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"routing-overlay\"" Oct 30 06:48:24.221908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.221891 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a0083e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a0083e\"" Oct 30 06:48:24.361768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.361719 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.362136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.362114 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd85l\" (UniqueName: \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-kube-api-access-gd85l\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.362283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.362269 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.362504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.362487 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.466645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.466540 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.467000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.466975 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd85l\" (UniqueName: \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-kube-api-access-gd85l\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.467150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.467134 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.467387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.467364 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.470205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.470143 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.483330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.478343 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.515331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.515177 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.523064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.523028 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd85l\" (UniqueName: \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-kube-api-access-gd85l\") pod \"payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.529248 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.529216 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:24.886726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:24.886693 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn"] Oct 30 06:48:24.888874 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:48:24.888850 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c7cf6b_7f60_4d47_8ebf_f6ada29eaacf.slice/crio-e859889359eb21f0152edc4d2e1d9a93c68e61c37cc406780c74623561944e4e WatchSource:0}: Error finding container e859889359eb21f0152edc4d2e1d9a93c68e61c37cc406780c74623561944e4e: Status 404 returned error can't find the container with id e859889359eb21f0152edc4d2e1d9a93c68e61c37cc406780c74623561944e4e Oct 30 06:48:25.182033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.181940 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528"] Oct 30 06:48:25.188963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.188670 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.191907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.191872 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" event={"ID":"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf","Type":"ContainerStarted","Data":"3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01"} Oct 30 06:48:25.194849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.194426 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" event={"ID":"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf","Type":"ContainerStarted","Data":"e859889359eb21f0152edc4d2e1d9a93c68e61c37cc406780c74623561944e4e"} Oct 30 06:48:25.194849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.194485 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:25.194849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.194505 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:25.194849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.194525 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" event={"ID":"f5969015-c270-4671-8af3-d5a565fa48a3","Type":"ContainerStarted","Data":"03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04"} Oct 30 06:48:25.195150 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.195093 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528"] Oct 30 06:48:25.204370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.203043 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"agc-cred-90dc2807c151\"" Oct 30 06:48:25.204370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.203177 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"payload-processing-external-model-e2e--7b857ee5-dockercfg-kqhvn\"" Oct 30 06:48:25.361939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.341628 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmvb7\" (UniqueName: \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-kube-api-access-qmvb7\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.362277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.362248 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.362781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.362754 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.371704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.371673 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.372041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.359015 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"routing-overlay\"" Oct 30 06:48:25.395386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.358727 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"openshift-service-ca.crt\"" Oct 30 06:48:25.395619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.358805 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f0145f\"" Oct 30 06:48:25.395726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.358922 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"kube-root-ca.crt\"" Oct 30 06:48:25.446397 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.428351 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" podStartSLOduration=1.428330806 podStartE2EDuration="1.428330806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:25.335627115 +0000 UTC m=+3319.331584138" watchObservedRunningTime="2026-10-30 06:48:25.428330806 +0000 UTC m=+3319.424287819" Oct 30 06:48:25.494328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.483514 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.494328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.483668 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.494328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.483818 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmvb7\" (UniqueName: \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-kube-api-access-qmvb7\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.494328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.483887 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.505355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.495638 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.521328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.519100 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.526353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.523904 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.533278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.532993 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmvb7\" (UniqueName: \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-kube-api-access-qmvb7\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:25.533278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.533139 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" podStartSLOduration=2.533124541 podStartE2EDuration="2.533124541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:25.416810947 +0000 UTC m=+3319.412767952" watchObservedRunningTime="2026-10-30 06:48:25.533124541 +0000 UTC m=+3319.529081552" Oct 30 06:48:25.814466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:25.814388 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:26.208418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:26.208372 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528"] Oct 30 06:48:26.213357 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:48:26.213287 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784b4a26_1aa6_4e2b_b626_c88deb312dc5.slice/crio-6b1b593a4da6baa285bd724a3066facdaa20d55582c4c15fdc1e9d11ebe782fd WatchSource:0}: Error finding container 6b1b593a4da6baa285bd724a3066facdaa20d55582c4c15fdc1e9d11ebe782fd: Status 404 returned error can't find the container with id 6b1b593a4da6baa285bd724a3066facdaa20d55582c4c15fdc1e9d11ebe782fd Oct 30 06:48:27.217607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:27.217568 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" event={"ID":"784b4a26-1aa6-4e2b-b626-c88deb312dc5","Type":"ContainerStarted","Data":"c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9"} Oct 30 06:48:27.218039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:27.217639 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" event={"ID":"784b4a26-1aa6-4e2b-b626-c88deb312dc5","Type":"ContainerStarted","Data":"6b1b593a4da6baa285bd724a3066facdaa20d55582c4c15fdc1e9d11ebe782fd"} Oct 30 06:48:27.218039 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:27.217668 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:27.279330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:27.264255 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" podStartSLOduration=2.2642352199999998 podStartE2EDuration="2.26423522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:27.256281537 +0000 UTC m=+3321.252238548" watchObservedRunningTime="2026-10-30 06:48:27.26423522 +0000 UTC m=+3321.260192232" Oct 30 06:48:28.220445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:28.220408 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528"] Oct 30 06:48:28.283143 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:28.283108 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f0145f/agc-cred-90dc2807c151: secret "agc-cred-90dc2807c151" not found Oct 30 06:48:28.283143 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:28.283139 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528: secret "agc-cred-90dc2807c151" not found Oct 30 06:48:28.283448 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:28.283207 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials podName:784b4a26-1aa6-4e2b-b626-c88deb312dc5 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:28.78318523 +0000 UTC m=+3322.779142219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5") : secret "agc-cred-90dc2807c151" not found Oct 30 06:48:28.283448 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:28.283226 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-plugins-e2e-extmod-a-f0145f: configmap "payload-processing-external-model-plugins-e2e-extmod-a-f0145f" not found Oct 30 06:48:28.283448 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:28.283271 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume podName:784b4a26-1aa6-4e2b-b626-c88deb312dc5 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:28.783261146 +0000 UTC m=+3322.779218148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-f0145f" not found Oct 30 06:48:28.797756 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:28.797722 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-plugins-e2e-extmod-a-f0145f: configmap "payload-processing-external-model-plugins-e2e-extmod-a-f0145f" not found Oct 30 06:48:28.798093 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:28.797807 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume podName:784b4a26-1aa6-4e2b-b626-c88deb312dc5 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:29.797786221 +0000 UTC m=+3323.793743212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-f0145f" not found Oct 30 06:48:28.798093 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:28.797913 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f0145f/agc-cred-90dc2807c151: secret "agc-cred-90dc2807c151" not found Oct 30 06:48:28.798093 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:28.797939 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528: secret "agc-cred-90dc2807c151" not found Oct 30 06:48:28.798093 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:28.797991 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials podName:784b4a26-1aa6-4e2b-b626-c88deb312dc5 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:29.797974634 +0000 UTC m=+3323.793931636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5") : secret "agc-cred-90dc2807c151" not found Oct 30 06:48:29.822184 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:29.822130 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f0145f/agc-cred-90dc2807c151: secret "agc-cred-90dc2807c151" not found Oct 30 06:48:29.822184 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:29.822172 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528: secret "agc-cred-90dc2807c151" not found Oct 30 06:48:29.822184 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:29.822174 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-plugins-e2e-extmod-a-f0145f: configmap "payload-processing-external-model-plugins-e2e-extmod-a-f0145f" not found Oct 30 06:48:29.822707 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:29.822247 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials podName:784b4a26-1aa6-4e2b-b626-c88deb312dc5 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:31.822223891 +0000 UTC m=+3325.818180903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5") : secret "agc-cred-90dc2807c151" not found Oct 30 06:48:29.822707 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:29.822267 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume podName:784b4a26-1aa6-4e2b-b626-c88deb312dc5 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:31.822257377 +0000 UTC m=+3325.818214381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-f0145f" not found Oct 30 06:48:31.263425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:31.263389 5001 generic.go:356] "Generic (PLEG): container finished" podID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerID="60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a" exitCode=0 Oct 30 06:48:31.263852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:31.263460 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" event={"ID":"3edcda5b-e672-43ae-9fdf-10b0a644d0ae","Type":"ContainerDied","Data":"60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a"} Oct 30 06:48:31.263953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:31.263939 5001 scope.go:122] "RemoveContainer" containerID="60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a" Oct 30 06:48:31.265869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:31.265835 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:48:31.266001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:31.265932 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:48:31.266001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:31.265980 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:48:31.854864 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:31.854816 5001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f0145f/agc-cred-90dc2807c151: secret "agc-cred-90dc2807c151" not found Oct 30 06:48:31.854864 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:31.854860 5001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528: secret "agc-cred-90dc2807c151" not found Oct 30 06:48:31.855115 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:31.854923 5001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-plugins-e2e-extmod-a-f0145f: configmap "payload-processing-external-model-plugins-e2e-extmod-a-f0145f" not found Oct 30 06:48:31.855115 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:31.854948 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials podName:784b4a26-1aa6-4e2b-b626-c88deb312dc5 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:35.854922296 +0000 UTC m=+3329.850879389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5") : secret "agc-cred-90dc2807c151" not found Oct 30 06:48:31.855115 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:31.854977 5001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume podName:784b4a26-1aa6-4e2b-b626-c88deb312dc5 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:35.854964971 +0000 UTC m=+3329.850921990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-f0145f" not found Oct 30 06:48:32.293389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:32.293246 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" event={"ID":"3edcda5b-e672-43ae-9fdf-10b0a644d0ae","Type":"ContainerStarted","Data":"c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c"} Oct 30 06:48:33.141673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:33.141642 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:48:33.141878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:33.141777 5001 patch_prober.go:29] interesting pod/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body= Oct 30 06:48:33.141878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:33.141818 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" Oct 30 06:48:34.141808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.141772 5001 patch_prober.go:29] interesting pod/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body= Oct 30 06:48:34.142233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.141834 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" Oct 30 06:48:34.243426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.243381 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" podUID="784b4a26-1aa6-4e2b-b626-c88deb312dc5" containerName="payload-processing" containerID="cri-o://c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9" gracePeriod=25 Oct 30 06:48:34.645073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.645045 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:34.828230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.828149 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume\") pod \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " Oct 30 06:48:34.828453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.828399 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-kube-api-access-qmvb7\" (UniqueName: \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-kube-api-access-qmvb7\") pod \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " Oct 30 06:48:34.828453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.828436 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-routing-overlay-volume\") pod \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " Oct 30 06:48:34.828599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.828489 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials\") pod \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\" (UID: \"784b4a26-1aa6-4e2b-b626-c88deb312dc5\") " Oct 30 06:48:34.828844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.828818 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-routing-overlay-volume" pod "784b4a26-1aa6-4e2b-b626-c88deb312dc5" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:48:34.828948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.828918 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume" pod "784b4a26-1aa6-4e2b-b626-c88deb312dc5" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:48:34.829100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.829085 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:48:34.829174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.829103 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784b4a26-1aa6-4e2b-b626-c88deb312dc5-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:48:34.830843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.830820 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-kube-api-access-qmvb7" pod "784b4a26-1aa6-4e2b-b626-c88deb312dc5" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5"). InnerVolumeSpecName "kube-api-access-qmvb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:48:34.830942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.830895 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials" pod "784b4a26-1aa6-4e2b-b626-c88deb312dc5" (UID: "784b4a26-1aa6-4e2b-b626-c88deb312dc5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:48:34.930563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.930526 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmvb7\" (UniqueName: \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-kube-api-access-qmvb7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:48:34.930563 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:34.930561 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/784b4a26-1aa6-4e2b-b626-c88deb312dc5-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:48:35.141712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:35.141669 5001 patch_prober.go:29] interesting pod/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body= Oct 30 06:48:35.141946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:35.141735 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" Oct 30 06:48:35.346279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:35.346245 5001 generic.go:356] "Generic (PLEG): container finished" podID="784b4a26-1aa6-4e2b-b626-c88deb312dc5" containerID="c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9" exitCode=0 Oct 30 06:48:35.346279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:35.346289 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" Oct 30 06:48:35.346579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:35.346326 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" event={"ID":"784b4a26-1aa6-4e2b-b626-c88deb312dc5","Type":"ContainerDied","Data":"c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9"} Oct 30 06:48:35.346579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:35.346379 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528" event={"ID":"784b4a26-1aa6-4e2b-b626-c88deb312dc5","Type":"ContainerDied","Data":"6b1b593a4da6baa285bd724a3066facdaa20d55582c4c15fdc1e9d11ebe782fd"} Oct 30 06:48:35.346579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:35.346409 5001 scope.go:122] "RemoveContainer" containerID="c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9" Oct 30 06:48:35.421164 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:35.421135 5001 scope.go:122] "RemoveContainer" containerID="c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9" Oct 30 06:48:35.421651 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:35.421612 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9\": container with ID starting with c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9 not found: ID does not exist" containerID="c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9" Oct 30 06:48:35.421789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:35.421662 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9"} err="failed to get container status \"c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9\": rpc error: code = NotFound desc = could not find container \"c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9\": container with ID starting with c2c45f6bba134057dba090bc66990f30b0fc985ccd09764a1a422433619999e9 not found: ID does not exist" Oct 30 06:48:35.449556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:35.449516 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528"] Oct 30 06:48:35.485035 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:35.483044 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-77ccd4528"] Oct 30 06:48:36.132201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.132162 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784b4a26-1aa6-4e2b-b626-c88deb312dc5" path="/var/lib/kubelet/pods/784b4a26-1aa6-4e2b-b626-c88deb312dc5/volumes" Oct 30 06:48:36.141514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.141487 5001 patch_prober.go:29] interesting pod/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body= Oct 30 06:48:36.141667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.141534 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" Oct 30 06:48:36.207907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.207870 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:48:36.208422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.207951 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:48:36.837510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.837476 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6"] Oct 30 06:48:36.840337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.840286 5001 cpu_manager.go:414] "containerMap: removing container" podUID="784b4a26-1aa6-4e2b-b626-c88deb312dc5" containerName="payload-processing" Oct 30 06:48:36.840571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.840552 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784b4a26-1aa6-4e2b-b626-c88deb312dc5" containerName="payload-processing" Oct 30 06:48:36.840982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.840960 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="784b4a26-1aa6-4e2b-b626-c88deb312dc5" containerName="payload-processing" Oct 30 06:48:36.919763 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.919729 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:36.922299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.922258 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6"] Oct 30 06:48:36.934246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:36.934223 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8baf7e-openshift-default-dockercfg-p8v7r\"" Oct 30 06:48:37.028372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.028294 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-credential-socket\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.028663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.028400 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istiod-ca-cert\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.028663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.028450 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-token\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.028663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.028593 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-socket\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.028829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.028719 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-podinfo\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.028829 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.028795 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-certs\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.028942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.028840 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-envoy\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.028942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.028912 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-data\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.029059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.028993 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf7wx\" (UniqueName: \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-kube-api-access-sf7wx\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.129940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.129903 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-credential-socket\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.130178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.129959 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istiod-ca-cert\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.130178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.129999 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-token\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.130178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.130086 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-socket\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.130411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.130200 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-podinfo\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.130411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.130256 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-certs\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.130411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.130349 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-envoy\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.130411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.130397 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-data\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.130637 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.130448 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf7wx\" (UniqueName: \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-kube-api-access-sf7wx\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.130931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.130897 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-certs\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.131121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.131074 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-data\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.131393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.131363 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-socket\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.131531 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.131509 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-credential-socket\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.131741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.131723 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istiod-ca-cert\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.133166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.133148 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-envoy\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.133607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.133588 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-podinfo\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.142575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.142552 5001 patch_prober.go:29] interesting pod/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body= Oct 30 06:48:37.142683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.142594 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" Oct 30 06:48:37.171330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.164026 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf7wx\" (UniqueName: \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-kube-api-access-sf7wx\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.171330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.164805 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-token\") pod \"e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.238893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.238856 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:37.692605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.678529 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6"] Oct 30 06:48:37.695948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.695890 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:48:37.696083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.696008 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:48:37.696083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.696056 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:48:37.828341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.828285 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" event={"ID":"65dfe1f4-528d-4b3d-9bf4-7a83d57130be","Type":"ContainerStarted","Data":"d04ed4c49dbc4f48669f06a06f13cb4c0e43f762aba221f9af6c3aa693d2e433"} Oct 30 06:48:37.828553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:37.828370 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" event={"ID":"65dfe1f4-528d-4b3d-9bf4-7a83d57130be","Type":"ContainerStarted","Data":"1413520afd8f37354950a4bc86eed9d9e785ef85adc3c0574ede712cfabf0cb0"} Oct 30 06:48:38.239425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:38.239145 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:38.240466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:38.240427 5001 patch_prober.go:29] interesting pod/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 06:48:38.240628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:38.240482 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 06:48:38.380332 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:48:38.379541 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:48:38.424800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:38.423536 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:48:38.425394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:38.425370 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:48:38.432787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:38.432756 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:48:38.469463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:38.469403 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" podStartSLOduration=2.469383099 podStartE2EDuration="2.469383099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:37.882727994 +0000 UTC m=+3331.878685006" watchObservedRunningTime="2026-10-30 06:48:38.469383099 +0000 UTC m=+3332.465340111" Oct 30 06:48:39.244830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:39.244795 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:39.850381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:39.850345 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:39.851473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:39.851445 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:48:42.186892 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.186855 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb"] Oct 30 06:48:42.191013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.190990 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:48:42.212170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.212143 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb"] Oct 30 06:48:42.216072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.216047 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8baf7e\"" Oct 30 06:48:42.216425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.216397 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8baf7e-dockercfg-rss5z\"" Oct 30 06:48:42.334331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.323074 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5qhc\" (UniqueName: \"kubernetes.io/projected/067fb315-b20e-42f3-b792-d081d921b529-kube-api-access-q5qhc\") pod \"payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb\" (UID: \"067fb315-b20e-42f3-b792-d081d921b529\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:48:42.334331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.323397 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067fb315-b20e-42f3-b792-d081d921b529-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb\" (UID: \"067fb315-b20e-42f3-b792-d081d921b529\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:48:42.430330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.418273 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr"] Oct 30 06:48:42.449587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.447745 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067fb315-b20e-42f3-b792-d081d921b529-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb\" (UID: \"067fb315-b20e-42f3-b792-d081d921b529\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:48:42.449587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.447941 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5qhc\" (UniqueName: \"kubernetes.io/projected/067fb315-b20e-42f3-b792-d081d921b529-kube-api-access-q5qhc\") pod \"payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb\" (UID: \"067fb315-b20e-42f3-b792-d081d921b529\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:48:42.479527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.479482 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067fb315-b20e-42f3-b792-d081d921b529-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb\" (UID: \"067fb315-b20e-42f3-b792-d081d921b529\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:48:42.495998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.495968 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5qhc\" (UniqueName: \"kubernetes.io/projected/067fb315-b20e-42f3-b792-d081d921b529-kube-api-access-q5qhc\") pod \"payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb\" (UID: \"067fb315-b20e-42f3-b792-d081d921b529\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:48:42.500902 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.500874 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:48:42.520396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.510592 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:48:42.600349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.570283 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr"] Oct 30 06:48:42.673606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.673571 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acd94f6d-738f-46ea-8974-084320983b78-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr\" (UID: \"acd94f6d-738f-46ea-8974-084320983b78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:48:42.673951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.673931 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvpv\" (UniqueName: \"kubernetes.io/projected/acd94f6d-738f-46ea-8974-084320983b78-kube-api-access-vqvpv\") pod \"payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr\" (UID: \"acd94f6d-738f-46ea-8974-084320983b78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:48:42.776045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.775999 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvpv\" (UniqueName: \"kubernetes.io/projected/acd94f6d-738f-46ea-8974-084320983b78-kube-api-access-vqvpv\") pod \"payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr\" (UID: \"acd94f6d-738f-46ea-8974-084320983b78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:48:42.776381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.776358 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acd94f6d-738f-46ea-8974-084320983b78-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr\" (UID: \"acd94f6d-738f-46ea-8974-084320983b78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:48:42.777488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.777364 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acd94f6d-738f-46ea-8974-084320983b78-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr\" (UID: \"acd94f6d-738f-46ea-8974-084320983b78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:48:42.802440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.801343 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvpv\" (UniqueName: \"kubernetes.io/projected/acd94f6d-738f-46ea-8974-084320983b78-kube-api-access-vqvpv\") pod \"payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr\" (UID: \"acd94f6d-738f-46ea-8974-084320983b78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:48:42.936559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.936522 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:48:42.959280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:42.959248 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb"] Oct 30 06:48:42.960824 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:48:42.960796 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067fb315_b20e_42f3_b792_d081d921b529.slice/crio-0787ebc307f5c0147904d48187d2d6615d0c9894bf9c6d4e792896efc3f3bda4 WatchSource:0}: Error finding container 0787ebc307f5c0147904d48187d2d6615d0c9894bf9c6d4e792896efc3f3bda4: Status 404 returned error can't find the container with id 0787ebc307f5c0147904d48187d2d6615d0c9894bf9c6d4e792896efc3f3bda4 Oct 30 06:48:43.261835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:43.261809 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr"] Oct 30 06:48:43.262800 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:48:43.262774 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd94f6d_738f_46ea_8974_084320983b78.slice/crio-e6ccb60bdb0ee55b669785de20be808cf4474342ca4c4cfec1f1842aae3724aa WatchSource:0}: Error finding container e6ccb60bdb0ee55b669785de20be808cf4474342ca4c4cfec1f1842aae3724aa: Status 404 returned error can't find the container with id e6ccb60bdb0ee55b669785de20be808cf4474342ca4c4cfec1f1842aae3724aa Oct 30 06:48:43.899641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:43.899585 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" event={"ID":"067fb315-b20e-42f3-b792-d081d921b529","Type":"ContainerStarted","Data":"753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575"} Oct 30 06:48:43.900573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:43.900546 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" event={"ID":"067fb315-b20e-42f3-b792-d081d921b529","Type":"ContainerStarted","Data":"0787ebc307f5c0147904d48187d2d6615d0c9894bf9c6d4e792896efc3f3bda4"} Oct 30 06:48:43.900733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:43.900706 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:48:43.901801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:43.901769 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" event={"ID":"acd94f6d-738f-46ea-8974-084320983b78","Type":"ContainerStarted","Data":"f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8"} Oct 30 06:48:43.901912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:43.901824 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" event={"ID":"acd94f6d-738f-46ea-8974-084320983b78","Type":"ContainerStarted","Data":"e6ccb60bdb0ee55b669785de20be808cf4474342ca4c4cfec1f1842aae3724aa"} Oct 30 06:48:43.901912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:43.901854 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:48:43.945186 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:43.945131 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" podStartSLOduration=1.945117135 podStartE2EDuration="1.945117135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:43.940891933 +0000 UTC m=+3337.936848943" watchObservedRunningTime="2026-10-30 06:48:43.945117135 +0000 UTC m=+3337.941074145" Oct 30 06:48:43.984827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:43.984765 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" podStartSLOduration=1.984745395 podStartE2EDuration="1.984745395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:43.98053366 +0000 UTC m=+3337.976490670" watchObservedRunningTime="2026-10-30 06:48:43.984745395 +0000 UTC m=+3337.980702406" Oct 30 06:48:49.628364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:49.623166 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:48:49.762331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:49.758409 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:48:50.156501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:50.156448 5001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Oct 30 06:48:50.295540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:50.295494 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:48:50.430899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:50.430798 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:48:55.007517 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:55.007480 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:48:56.025612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:56.025585 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:48:57.353795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:57.353704 5001 generic.go:356] "Generic (PLEG): container finished" podID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerID="cab136d2978bfa28648e8d879fe56ef5fc787f4c97ff1794c610f8ece6d28496" exitCode=0 Oct 30 06:48:57.353795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:57.353776 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" event={"ID":"0c7e9f40-f353-436a-b771-ea6223ac02db","Type":"ContainerDied","Data":"cab136d2978bfa28648e8d879fe56ef5fc787f4c97ff1794c610f8ece6d28496"} Oct 30 06:48:57.354449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:57.354296 5001 scope.go:122] "RemoveContainer" containerID="cab136d2978bfa28648e8d879fe56ef5fc787f4c97ff1794c610f8ece6d28496" Oct 30 06:48:57.356493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:57.356458 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:48:57.356630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:57.356532 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:48:57.356630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:57.356564 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:48:58.405676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:58.405640 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" event={"ID":"0c7e9f40-f353-436a-b771-ea6223ac02db","Type":"ContainerStarted","Data":"6c97f7ed9a4f9985c6d94e61802b535a2c777f871e4d10c2a3e58223849bf71d"} Oct 30 06:48:59.357341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:59.355607 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:48:59.357341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:59.355718 5001 patch_prober.go:29] interesting pod/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.144:15021/healthz/ready\": dial tcp 10.132.1.144:15021: connect: connection refused" start-of-body= Oct 30 06:48:59.357341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:48:59.355767 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.144:15021/healthz/ready\": dial tcp 10.132.1.144:15021: connect: connection refused" Oct 30 06:49:00.362335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:00.360426 5001 patch_prober.go:29] interesting pod/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.144:15021/healthz/ready\": dial tcp 10.132.1.144:15021: connect: connection refused" start-of-body= Oct 30 06:49:00.362335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:00.360494 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.144:15021/healthz/ready\": dial tcp 10.132.1.144:15021: connect: connection refused" Oct 30 06:49:01.352777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:01.352735 5001 patch_prober.go:29] interesting pod/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.144:15021/healthz/ready\": dial tcp 10.132.1.144:15021: connect: connection refused" start-of-body= Oct 30 06:49:01.353111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:01.353076 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.144:15021/healthz/ready\": dial tcp 10.132.1.144:15021: connect: connection refused" Oct 30 06:49:02.366332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:02.353751 5001 patch_prober.go:29] interesting pod/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.144:15021/healthz/ready\": dial tcp 10.132.1.144:15021: connect: connection refused" start-of-body= Oct 30 06:49:02.366332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:02.353818 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.144:15021/healthz/ready\": dial tcp 10.132.1.144:15021: connect: connection refused" Oct 30 06:49:03.356334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:03.355710 5001 patch_prober.go:29] interesting pod/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.144:15021/healthz/ready\": dial tcp 10.132.1.144:15021: connect: connection refused" start-of-body= Oct 30 06:49:03.356334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:03.355789 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.144:15021/healthz/ready\": dial tcp 10.132.1.144:15021: connect: connection refused" Oct 30 06:49:04.360828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.360788 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:49:04.362285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.362268 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:49:04.363041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.363018 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:49:04.559340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.557070 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z"] Oct 30 06:49:04.562747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.562722 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.591251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.591196 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z"] Oct 30 06:49:04.687485 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.684872 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-89f101-openshift-default-dockercfg-j8xrw\"" Oct 30 06:49:04.710283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.708389 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfhm\" (UniqueName: \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-kube-api-access-twfhm\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.710283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.708520 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-socket\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.710283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.708566 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-credential-socket\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.710283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.708599 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-certs\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.710283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.708628 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-podinfo\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.710283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.708676 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29524a8b-436b-4b95-a88c-8b4d433474b1-istiod-ca-cert\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.710283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.708718 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-token\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.710283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.708958 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-data\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.710283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.708986 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-envoy\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.814332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.813458 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-data\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.814332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.813510 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-envoy\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.814332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.813593 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfhm\" (UniqueName: \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-kube-api-access-twfhm\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.814332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.813686 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-socket\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.814332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.813728 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-credential-socket\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.814332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.813758 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-certs\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.814332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.813788 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-podinfo\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.814332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.813835 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29524a8b-436b-4b95-a88c-8b4d433474b1-istiod-ca-cert\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.814332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.813874 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-token\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.814919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.814604 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-data\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.826335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.822942 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-credential-socket\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.826335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.823928 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-socket\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.826335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.824626 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29524a8b-436b-4b95-a88c-8b4d433474b1-istiod-ca-cert\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.826335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.824884 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-certs\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.832623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.828840 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-envoy\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.915332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.913637 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-podinfo\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.915332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.914145 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-token\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:04.952381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:04.948117 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfhm\" (UniqueName: \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-kube-api-access-twfhm\") pod \"e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:05.210945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:05.210857 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:05.543621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:05.543589 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z"] Oct 30 06:49:05.545182 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:49:05.545160 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29524a8b_436b_4b95_a88c_8b4d433474b1.slice/crio-ede6fde7f0d2184ca064ed963610d012e849dd1abcc363da69b7108b09161fe4 WatchSource:0}: Error finding container ede6fde7f0d2184ca064ed963610d012e849dd1abcc363da69b7108b09161fe4: Status 404 returned error can't find the container with id ede6fde7f0d2184ca064ed963610d012e849dd1abcc363da69b7108b09161fe4 Oct 30 06:49:05.548111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:05.548074 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:49:05.548211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:05.548145 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:49:05.548211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:05.548176 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:49:06.520182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:06.520143 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" event={"ID":"29524a8b-436b-4b95-a88c-8b4d433474b1","Type":"ContainerStarted","Data":"3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9"} Oct 30 06:49:06.520393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:06.520244 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" event={"ID":"29524a8b-436b-4b95-a88c-8b4d433474b1","Type":"ContainerStarted","Data":"ede6fde7f0d2184ca064ed963610d012e849dd1abcc363da69b7108b09161fe4"} Oct 30 06:49:06.574968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:06.574916 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" podStartSLOduration=2.574897922 podStartE2EDuration="2.574897922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:49:06.573340216 +0000 UTC m=+3360.569297233" watchObservedRunningTime="2026-10-30 06:49:06.574897922 +0000 UTC m=+3360.570854932" Oct 30 06:49:07.211782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:07.211750 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:07.218580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:07.218553 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:07.535413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:07.534348 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:07.535625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:07.535540 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:49:08.268229 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:49:08.268194 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:49:10.942277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:10.942231 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv"] Oct 30 06:49:10.948595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:10.948569 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:49:10.965627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:10.965577 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv"] Oct 30 06:49:10.969201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:10.969171 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-89f101-dockercfg-4t4pl\"" Oct 30 06:49:11.032331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:11.029773 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-89f101\"" Oct 30 06:49:11.066341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:11.065511 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0711132a-6143-43f8-91a8-979b147a168d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv\" (UID: \"0711132a-6143-43f8-91a8-979b147a168d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:49:11.066341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:11.065788 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7r9z\" (UniqueName: \"kubernetes.io/projected/0711132a-6143-43f8-91a8-979b147a168d-kube-api-access-g7r9z\") pod \"payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv\" (UID: \"0711132a-6143-43f8-91a8-979b147a168d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:49:11.181331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:11.177277 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0711132a-6143-43f8-91a8-979b147a168d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv\" (UID: \"0711132a-6143-43f8-91a8-979b147a168d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:49:11.181331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:11.177592 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7r9z\" (UniqueName: \"kubernetes.io/projected/0711132a-6143-43f8-91a8-979b147a168d-kube-api-access-g7r9z\") pod \"payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv\" (UID: \"0711132a-6143-43f8-91a8-979b147a168d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:49:11.270173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:11.270090 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7r9z\" (UniqueName: \"kubernetes.io/projected/0711132a-6143-43f8-91a8-979b147a168d-kube-api-access-g7r9z\") pod \"payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv\" (UID: \"0711132a-6143-43f8-91a8-979b147a168d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:49:11.270350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:11.270208 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0711132a-6143-43f8-91a8-979b147a168d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv\" (UID: \"0711132a-6143-43f8-91a8-979b147a168d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:49:11.566557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:11.566468 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:49:11.889798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:11.889770 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv"] Oct 30 06:49:11.892728 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:49:11.892697 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0711132a_6143_43f8_91a8_979b147a168d.slice/crio-0dafeb23fe5946bb3c9e92b0314491111cb1a133801f0891061442ed5245b48d WatchSource:0}: Error finding container 0dafeb23fe5946bb3c9e92b0314491111cb1a133801f0891061442ed5245b48d: Status 404 returned error can't find the container with id 0dafeb23fe5946bb3c9e92b0314491111cb1a133801f0891061442ed5245b48d Oct 30 06:49:12.039222 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:12.039179 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7"] Oct 30 06:49:12.049471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:12.049439 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25598b7b-ww7m7"] Oct 30 06:49:12.121618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:12.121579 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f2b5ed-ff84-40df-9ad7-a1ff3586eecb" path="/var/lib/kubelet/pods/71f2b5ed-ff84-40df-9ad7-a1ff3586eecb/volumes" Oct 30 06:49:12.596483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:12.596451 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" event={"ID":"0711132a-6143-43f8-91a8-979b147a168d","Type":"ContainerStarted","Data":"f45195ccdf2322bafead410660548b17d383f43ca4b6aa79098ef51a84c09a51"} Oct 30 06:49:12.596714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:12.596500 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" event={"ID":"0711132a-6143-43f8-91a8-979b147a168d","Type":"ContainerStarted","Data":"0dafeb23fe5946bb3c9e92b0314491111cb1a133801f0891061442ed5245b48d"} Oct 30 06:49:12.596714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:12.596558 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:49:12.635713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:12.635662 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" podStartSLOduration=2.635648313 podStartE2EDuration="2.635648313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:49:12.634448829 +0000 UTC m=+3366.630405838" watchObservedRunningTime="2026-10-30 06:49:12.635648313 +0000 UTC m=+3366.631605322" Oct 30 06:49:23.608964 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:23.608871 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:49:27.393334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:27.392549 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:49:36.497093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.497058 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd"] Oct 30 06:49:36.505726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.505691 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.529592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.529558 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a5f459-openshift-default-dockercfg-b4d9s\"" Oct 30 06:49:36.532073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.532032 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd"] Oct 30 06:49:36.572200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.572149 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-istio-token\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.572487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.572338 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-envoy\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.572487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.572360 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313ac877-7f04-420f-a158-32c97424341a-istiod-ca-cert\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.572641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.572488 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-certs\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.572641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.572521 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313ac877-7f04-420f-a158-32c97424341a-istio-podinfo\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.572641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.572599 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-socket\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.572821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.572656 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfm6\" (UniqueName: \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-kube-api-access-plfm6\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.572821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.572795 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-data\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.572934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.572832 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-credential-socket\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.750391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.721670 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-data\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.750391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.721739 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-credential-socket\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.750391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.721849 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-istio-token\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.750391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.721997 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-envoy\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.750391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.722024 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313ac877-7f04-420f-a158-32c97424341a-istiod-ca-cert\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.750391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.722160 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-certs\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.750391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.722190 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313ac877-7f04-420f-a158-32c97424341a-istio-podinfo\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.750391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.722261 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-socket\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.750391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.722347 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfm6\" (UniqueName: \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-kube-api-access-plfm6\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.750391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.723218 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-data\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.750391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.723505 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-credential-socket\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.785343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.769123 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313ac877-7f04-420f-a158-32c97424341a-istiod-ca-cert\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.785343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.769403 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-certs\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.822557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.815694 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-socket\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.822557 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.815890 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-envoy\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.852430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.828748 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfm6\" (UniqueName: \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-kube-api-access-plfm6\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.858425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.858136 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313ac877-7f04-420f-a158-32c97424341a-istio-podinfo\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:36.858425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:36.858380 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-istio-token\") pod \"e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:37.138657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:37.138615 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:37.487768 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:37.487739 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd"] Oct 30 06:49:37.490566 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:49:37.490532 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313ac877_7f04_420f_a158_32c97424341a.slice/crio-0884a02c0c57a8b43ea08c03958d771899e2a67f889c5b38b616dd8b3603a35f WatchSource:0}: Error finding container 0884a02c0c57a8b43ea08c03958d771899e2a67f889c5b38b616dd8b3603a35f: Status 404 returned error can't find the container with id 0884a02c0c57a8b43ea08c03958d771899e2a67f889c5b38b616dd8b3603a35f Oct 30 06:49:37.494109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:37.494070 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:49:37.494237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:37.494175 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:49:37.494237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:37.494223 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:49:37.965025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:37.964976 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" event={"ID":"313ac877-7f04-420f-a158-32c97424341a","Type":"ContainerStarted","Data":"569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1"} Oct 30 06:49:37.965566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:37.965091 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" event={"ID":"313ac877-7f04-420f-a158-32c97424341a","Type":"ContainerStarted","Data":"0884a02c0c57a8b43ea08c03958d771899e2a67f889c5b38b616dd8b3603a35f"} Oct 30 06:49:38.011675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:38.011616 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" podStartSLOduration=2.01159851 podStartE2EDuration="2.01159851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:49:38.007952352 +0000 UTC m=+3392.003909363" watchObservedRunningTime="2026-10-30 06:49:38.01159851 +0000 UTC m=+3392.007555519" Oct 30 06:49:38.148432 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:38.148397 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:38.188004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:38.187967 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:38.286712 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:49:38.286522 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:49:38.978139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:38.978098 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:38.979394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:38.979368 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:49:41.004466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:41.004434 5001 generic.go:356] "Generic (PLEG): container finished" podID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerID="f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df" exitCode=0 Oct 30 06:49:41.004951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:41.004511 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" event={"ID":"06bbf981-0119-4ab2-9a8d-51e21306a98d","Type":"ContainerDied","Data":"f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df"} Oct 30 06:49:41.005190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:41.005172 5001 scope.go:122] "RemoveContainer" containerID="f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df" Oct 30 06:49:41.007261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:41.007230 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:49:41.007373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:41.007324 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:49:41.007373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:41.007361 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:49:42.016979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:42.016945 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" event={"ID":"06bbf981-0119-4ab2-9a8d-51e21306a98d","Type":"ContainerStarted","Data":"33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77"} Oct 30 06:49:42.836337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:42.835537 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:49:42.843423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:42.843376 5001 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 30 06:49:42.843739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:42.843689 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 30 06:49:43.828598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:43.828566 5001 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 30 06:49:43.829044 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:43.828621 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 30 06:49:44.829086 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:44.829051 5001 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 30 06:49:44.829569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:44.829110 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 30 06:49:45.829236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:45.829198 5001 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 30 06:49:45.829808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:45.829265 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 30 06:49:46.829171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:46.829133 5001 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 30 06:49:46.829389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:46.829198 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 30 06:49:47.032702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:47.032662 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb"] Oct 30 06:49:47.043819 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:47.043786 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25fb5c9-2w6rb"] Oct 30 06:49:47.390393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:47.390354 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:49:47.828825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:47.828739 5001 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 30 06:49:47.828825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:47.828804 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 30 06:49:48.124535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:48.124493 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595bb0e9-db79-4e9e-9a95-752eb8435718" path="/var/lib/kubelet/pods/595bb0e9-db79-4e9e-9a95-752eb8435718/volumes" Oct 30 06:49:48.834283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:48.834254 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:49:48.835336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:48.835296 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:49:57.033026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:57.032994 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc"] Oct 30 06:49:57.043860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:57.043827 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae34f-f7wmc"] Oct 30 06:49:58.120626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:49:58.120593 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4345edf9-316c-4034-9bd2-9daba4b25b10" path="/var/lib/kubelet/pods/4345edf9-316c-4034-9bd2-9daba4b25b10/volumes" Oct 30 06:50:07.419137 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:07.419102 5001 generic.go:356] "Generic (PLEG): container finished" podID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerID="d04ed4c49dbc4f48669f06a06f13cb4c0e43f762aba221f9af6c3aa693d2e433" exitCode=0 Oct 30 06:50:07.419811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:07.419183 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" event={"ID":"65dfe1f4-528d-4b3d-9bf4-7a83d57130be","Type":"ContainerDied","Data":"d04ed4c49dbc4f48669f06a06f13cb4c0e43f762aba221f9af6c3aa693d2e433"} Oct 30 06:50:07.419811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:07.419786 5001 scope.go:122] "RemoveContainer" containerID="d04ed4c49dbc4f48669f06a06f13cb4c0e43f762aba221f9af6c3aa693d2e433" Oct 30 06:50:07.421875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:07.421846 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:50:07.421973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:07.421918 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:50:07.421973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:07.421951 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:50:08.270324 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:50:08.270269 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:50:08.443335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:08.442864 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" event={"ID":"65dfe1f4-528d-4b3d-9bf4-7a83d57130be","Type":"ContainerStarted","Data":"8637ee5f1fca0d2f4c2c8ff8d465834513b894929972632909ad99899eccc68d"} Oct 30 06:50:09.239372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:09.239285 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:50:09.240391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:09.240292 5001 patch_prober.go:29] interesting pod/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 06:50:09.240516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:09.240396 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 06:50:09.850056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:09.850018 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:50:10.239443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:10.239403 5001 patch_prober.go:29] interesting pod/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 06:50:10.239641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:10.239459 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 06:50:11.239382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:11.239297 5001 patch_prober.go:29] interesting pod/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 06:50:11.239844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:11.239426 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 06:50:11.700666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:11.700634 5001 scope.go:122] "RemoveContainer" containerID="8f4d7c4ecaa5ed0dca9f1fb0bd2a8fd6da665eb8598328e3952f037e87aeb918" Oct 30 06:50:12.239237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:12.239205 5001 patch_prober.go:29] interesting pod/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 06:50:12.239446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:12.239267 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 06:50:13.239988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:13.239954 5001 patch_prober.go:29] interesting pod/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Oct 30 06:50:13.240413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:13.240010 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Oct 30 06:50:14.244924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:14.244890 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:50:14.245953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:14.245934 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:50:32.223356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:32.221388 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:50:33.997462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:33.997425 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p"] Oct 30 06:50:34.004770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.004734 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.061403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.061066 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-be48b4-openshift-default-dockercfg-lmxxb\"" Oct 30 06:50:34.094258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.093088 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdrw\" (UniqueName: \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-kube-api-access-rpdrw\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.094258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.093152 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-envoy\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.094258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.093261 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-token\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.094258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.093475 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-podinfo\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.094258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.093678 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-certs\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.094258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.093792 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-data\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.094258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.093865 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-socket\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.094258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.093897 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-credential-socket\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.094258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.093925 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istiod-ca-cert\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.099395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.099360 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p"] Oct 30 06:50:34.201571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.201527 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-certs\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.213334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.206605 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-data\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.213689 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-socket\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.234436 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-credential-socket\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.234514 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istiod-ca-cert\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.234621 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdrw\" (UniqueName: \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-kube-api-access-rpdrw\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.234668 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-envoy\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.234773 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-token\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.234898 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-podinfo\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.235496 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-certs\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.235761 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-data\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.235994 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-socket\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.236218 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-credential-socket\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.237330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.236868 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istiod-ca-cert\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.500321 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.500267 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-envoy\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.551380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.538590 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdrw\" (UniqueName: \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-kube-api-access-rpdrw\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.694519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.694484 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-token\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.739331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.736683 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-podinfo\") pod \"e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:34.980574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:34.980532 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:35.315031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:35.314999 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p"] Oct 30 06:50:35.315801 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:50:35.315775 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd760fe2_e8c7_42f9_8a3f_40ec810159b4.slice/crio-cc112da958e4a220e7451554181e1fc55010f3832a8e8dac59b278bbb4158730 WatchSource:0}: Error finding container cc112da958e4a220e7451554181e1fc55010f3832a8e8dac59b278bbb4158730: Status 404 returned error can't find the container with id cc112da958e4a220e7451554181e1fc55010f3832a8e8dac59b278bbb4158730 Oct 30 06:50:35.319010 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:35.318975 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:50:35.319100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:35.319051 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:50:35.319100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:35.319086 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:50:35.987334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:35.986765 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" event={"ID":"cd760fe2-e8c7-42f9-8a3f-40ec810159b4","Type":"ContainerStarted","Data":"5a2371cec333f16b59f9054c7132a50bd8827237c999ae5f964b7d337f57a303"} Oct 30 06:50:35.987334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:35.986894 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" event={"ID":"cd760fe2-e8c7-42f9-8a3f-40ec810159b4","Type":"ContainerStarted","Data":"cc112da958e4a220e7451554181e1fc55010f3832a8e8dac59b278bbb4158730"} Oct 30 06:50:36.037048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:36.036981 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" podStartSLOduration=3.036962206 podStartE2EDuration="3.036962206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:50:36.031754523 +0000 UTC m=+3450.027711534" watchObservedRunningTime="2026-10-30 06:50:36.036962206 +0000 UTC m=+3450.032919217" Oct 30 06:50:36.989380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:36.988668 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:36.998365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:36.998288 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:37.002604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:37.002576 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:37.003555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:37.003530 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:50:38.296558 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:50:38.296520 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:50:40.479722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:40.479682 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr"] Oct 30 06:50:40.484293 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:40.484269 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:50:40.507584 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:40.507541 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr"] Oct 30 06:50:40.511789 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:40.511763 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-be48b4-dockercfg-nh9zn\"" Oct 30 06:50:40.512092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:40.511827 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-be48b4\"" Oct 30 06:50:40.922901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:40.922812 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nlrp\" (UniqueName: \"kubernetes.io/projected/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-kube-api-access-4nlrp\") pod \"payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr\" (UID: \"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:50:40.923251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:40.923222 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr\" (UID: \"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:50:41.025381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:41.025338 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nlrp\" (UniqueName: \"kubernetes.io/projected/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-kube-api-access-4nlrp\") pod \"payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr\" (UID: \"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:50:41.025591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:41.025574 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr\" (UID: \"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:50:41.026284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:41.026264 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr\" (UID: \"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:50:41.048569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:41.048529 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nlrp\" (UniqueName: \"kubernetes.io/projected/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-kube-api-access-4nlrp\") pod \"payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr\" (UID: \"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:50:41.112558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:41.112523 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:50:41.438688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:41.438654 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr"] Oct 30 06:50:41.440204 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:50:41.440174 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30a5651_954e_4a58_b0a0_c6e7ba9d1ae5.slice/crio-ff6e20ef36caf08ee9a5d8dff5e25aec2196e068ea85129c0cb78810ef7a19d6 WatchSource:0}: Error finding container ff6e20ef36caf08ee9a5d8dff5e25aec2196e068ea85129c0cb78810ef7a19d6: Status 404 returned error can't find the container with id ff6e20ef36caf08ee9a5d8dff5e25aec2196e068ea85129c0cb78810ef7a19d6 Oct 30 06:50:42.057011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:42.056967 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" event={"ID":"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5","Type":"ContainerStarted","Data":"f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b"} Oct 30 06:50:42.057603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:42.057033 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" event={"ID":"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5","Type":"ContainerStarted","Data":"ff6e20ef36caf08ee9a5d8dff5e25aec2196e068ea85129c0cb78810ef7a19d6"} Oct 30 06:50:42.057603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:42.057061 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:50:42.100796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:42.100731 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" podStartSLOduration=2.100715935 podStartE2EDuration="2.100715935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:50:42.099049547 +0000 UTC m=+3456.095006563" watchObservedRunningTime="2026-10-30 06:50:42.100715935 +0000 UTC m=+3456.096672945" Oct 30 06:50:47.112705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:47.112618 5001 generic.go:356] "Generic (PLEG): container finished" podID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerID="3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9" exitCode=0 Oct 30 06:50:47.112705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:47.112696 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" event={"ID":"29524a8b-436b-4b95-a88c-8b4d433474b1","Type":"ContainerDied","Data":"3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9"} Oct 30 06:50:47.113190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:47.113029 5001 scope.go:122] "RemoveContainer" containerID="3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9" Oct 30 06:50:47.114958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:47.114930 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:50:47.115036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:47.115001 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:50:47.115036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:47.115032 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:50:48.124233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:48.124196 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" event={"ID":"29524a8b-436b-4b95-a88c-8b4d433474b1","Type":"ContainerStarted","Data":"0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9"} Oct 30 06:50:48.211181 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:48.211138 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:50:48.212394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:48.212364 5001 patch_prober.go:29] interesting pod/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 06:50:48.212542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:48.212413 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 06:50:49.212134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:49.212094 5001 patch_prober.go:29] interesting pod/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 06:50:49.212667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:49.212157 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 06:50:50.211282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:50.211249 5001 patch_prober.go:29] interesting pod/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 06:50:50.211484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:50.211299 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 06:50:51.216826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:51.216789 5001 patch_prober.go:29] interesting pod/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 06:50:51.217562 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:51.217478 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 06:50:52.212122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:52.212078 5001 patch_prober.go:29] interesting pod/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 06:50:52.212394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:52.212165 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 06:50:52.534761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:52.534674 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:50:53.098328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:53.097914 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:50:53.211560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:53.211525 5001 patch_prober.go:29] interesting pod/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" start-of-body= Oct 30 06:50:53.211752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:53.211599 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.159:15021/healthz/ready\": dial tcp 10.132.1.159:15021: connect: connection refused" Oct 30 06:50:54.217341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:54.217274 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:50:54.218654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:50:54.218630 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:51:08.276064 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:51:08.275997 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:51:10.019495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:10.019453 5001 patch_prober.go:29] interesting pod/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:51:10.019942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:10.019537 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:51:11.735125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:11.735091 5001 scope.go:122] "RemoveContainer" containerID="1d2419662c93b0a81b8c25928dc14d08302a16f95eb83d72c29e344b37312936" Oct 30 06:51:11.746812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:11.746790 5001 scope.go:122] "RemoveContainer" containerID="1f78947dae3abd4f94bea205139782befac6c8a5a37a7d392364d597ae995e4a" Oct 30 06:51:13.547110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:13.547079 5001 generic.go:356] "Generic (PLEG): container finished" podID="313ac877-7f04-420f-a158-32c97424341a" containerID="569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1" exitCode=0 Oct 30 06:51:13.547597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:13.547159 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" event={"ID":"313ac877-7f04-420f-a158-32c97424341a","Type":"ContainerDied","Data":"569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1"} Oct 30 06:51:13.547754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:13.547738 5001 scope.go:122] "RemoveContainer" containerID="569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1" Oct 30 06:51:13.549985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:13.549950 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:51:13.550060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:13.550035 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:51:13.550099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:13.550069 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:51:14.636366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:14.633714 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" event={"ID":"313ac877-7f04-420f-a158-32c97424341a","Type":"ContainerStarted","Data":"82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315"} Oct 30 06:51:15.139256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:15.139218 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:51:15.139519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:15.139482 5001 patch_prober.go:29] interesting pod/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.161:15021/healthz/ready\": dial tcp 10.132.1.161:15021: connect: connection refused" start-of-body= Oct 30 06:51:15.139607 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:15.139557 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.161:15021/healthz/ready\": dial tcp 10.132.1.161:15021: connect: connection refused" Oct 30 06:51:16.140005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:16.139972 5001 patch_prober.go:29] interesting pod/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.161:15021/healthz/ready\": dial tcp 10.132.1.161:15021: connect: connection refused" start-of-body= Oct 30 06:51:16.140453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:16.140035 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.161:15021/healthz/ready\": dial tcp 10.132.1.161:15021: connect: connection refused" Oct 30 06:51:17.139919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:17.139879 5001 patch_prober.go:29] interesting pod/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.161:15021/healthz/ready\": dial tcp 10.132.1.161:15021: connect: connection refused" start-of-body= Oct 30 06:51:17.140117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:17.139939 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.161:15021/healthz/ready\": dial tcp 10.132.1.161:15021: connect: connection refused" Oct 30 06:51:18.140569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:18.140529 5001 patch_prober.go:29] interesting pod/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.161:15021/healthz/ready\": dial tcp 10.132.1.161:15021: connect: connection refused" start-of-body= Oct 30 06:51:18.141179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:18.140608 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.161:15021/healthz/ready\": dial tcp 10.132.1.161:15021: connect: connection refused" Oct 30 06:51:19.139147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:19.139110 5001 patch_prober.go:29] interesting pod/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.161:15021/healthz/ready\": dial tcp 10.132.1.161:15021: connect: connection refused" start-of-body= Oct 30 06:51:19.139399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:19.139185 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.161:15021/healthz/ready\": dial tcp 10.132.1.161:15021: connect: connection refused" Oct 30 06:51:20.144868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:20.144834 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:51:20.145402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:20.145335 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:51:20.146026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:20.146008 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:51:36.908987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:36.908948 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:51:38.275360 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:51:38.275319 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:51:42.393398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.393354 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv"] Oct 30 06:51:42.397929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.397903 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:42.415747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.415721 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv"] Oct 30 06:51:42.419280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.419245 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:51:42.419438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.419299 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:51:42.419438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.419299 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:51:42.443906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.443859 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:51:42.489040 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.489004 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4s6\" (UniqueName: \"kubernetes.io/projected/db4038d3-ac1b-48ec-a31e-886ae343309c-kube-api-access-mm4s6\") pod \"maas-api-revoke-sub-bb49c2ad-cczkv\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:42.489265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.489096 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb49c2ad-cczkv\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:42.489374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.489347 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb49c2ad-cczkv\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:42.590704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.590668 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4s6\" (UniqueName: \"kubernetes.io/projected/db4038d3-ac1b-48ec-a31e-886ae343309c-kube-api-access-mm4s6\") pod \"maas-api-revoke-sub-bb49c2ad-cczkv\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:42.590921 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.590760 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb49c2ad-cczkv\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:42.590986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.590939 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb49c2ad-cczkv\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:42.591458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.591439 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb49c2ad-cczkv\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:42.593590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.593565 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb49c2ad-cczkv\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:42.614245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.614218 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4s6\" (UniqueName: \"kubernetes.io/projected/db4038d3-ac1b-48ec-a31e-886ae343309c-kube-api-access-mm4s6\") pod \"maas-api-revoke-sub-bb49c2ad-cczkv\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:42.714697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:42.714614 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:43.026814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:43.026778 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv"] Oct 30 06:51:43.028036 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:51:43.028004 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4038d3_ac1b_48ec_a31e_886ae343309c.slice/crio-eea45c6a10d920b89a1d947f7b831804db5522fc6a6945d02cda9728e9b5fbb5 WatchSource:0}: Error finding container eea45c6a10d920b89a1d947f7b831804db5522fc6a6945d02cda9728e9b5fbb5: Status 404 returned error can't find the container with id eea45c6a10d920b89a1d947f7b831804db5522fc6a6945d02cda9728e9b5fbb5 Oct 30 06:51:43.071399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:43.071370 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" event={"ID":"db4038d3-ac1b-48ec-a31e-886ae343309c","Type":"ContainerStarted","Data":"eea45c6a10d920b89a1d947f7b831804db5522fc6a6945d02cda9728e9b5fbb5"} Oct 30 06:51:44.082350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:44.082314 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" event={"ID":"db4038d3-ac1b-48ec-a31e-886ae343309c","Type":"ContainerStarted","Data":"fc00c777fdd4c8fa0449765de51f013a18136d3b4560ac043e5cdc2c333a31fa"} Oct 30 06:51:44.125038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:44.124991 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" podStartSLOduration=2.124976715 podStartE2EDuration="2.124976715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:51:44.120907301 +0000 UTC m=+3518.116864312" watchObservedRunningTime="2026-10-30 06:51:44.124976715 +0000 UTC m=+3518.120933761" Oct 30 06:51:45.094257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:45.094209 5001 generic.go:356] "Generic (PLEG): container finished" podID="db4038d3-ac1b-48ec-a31e-886ae343309c" containerID="fc00c777fdd4c8fa0449765de51f013a18136d3b4560ac043e5cdc2c333a31fa" exitCode=0 Oct 30 06:51:45.094738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:45.094294 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" event={"ID":"db4038d3-ac1b-48ec-a31e-886ae343309c","Type":"ContainerDied","Data":"fc00c777fdd4c8fa0449765de51f013a18136d3b4560ac043e5cdc2c333a31fa"} Oct 30 06:51:46.317880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:46.317854 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:46.468425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:46.468326 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-openshift-ca\") pod \"db4038d3-ac1b-48ec-a31e-886ae343309c\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " Oct 30 06:51:46.468594 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:46.468515 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-xks-ca\") pod \"db4038d3-ac1b-48ec-a31e-886ae343309c\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " Oct 30 06:51:46.468705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:46.468685 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-openshift-ca" pod "db4038d3-ac1b-48ec-a31e-886ae343309c" (UID: "db4038d3-ac1b-48ec-a31e-886ae343309c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:51:46.468784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:46.468769 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4038d3-ac1b-48ec-a31e-886ae343309c-kube-api-access-mm4s6\" (UniqueName: \"kubernetes.io/projected/db4038d3-ac1b-48ec-a31e-886ae343309c-kube-api-access-mm4s6\") pod \"db4038d3-ac1b-48ec-a31e-886ae343309c\" (UID: \"db4038d3-ac1b-48ec-a31e-886ae343309c\") " Oct 30 06:51:46.469596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:46.469569 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:51:46.470867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:46.470846 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-xks-ca" pod "db4038d3-ac1b-48ec-a31e-886ae343309c" (UID: "db4038d3-ac1b-48ec-a31e-886ae343309c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:51:46.471087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:46.471074 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4038d3-ac1b-48ec-a31e-886ae343309c-kube-api-access-mm4s6" pod "db4038d3-ac1b-48ec-a31e-886ae343309c" (UID: "db4038d3-ac1b-48ec-a31e-886ae343309c"). InnerVolumeSpecName "kube-api-access-mm4s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:51:46.572139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:46.572095 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4038d3-ac1b-48ec-a31e-886ae343309c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:51:46.572139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:46.572139 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4s6\" (UniqueName: \"kubernetes.io/projected/db4038d3-ac1b-48ec-a31e-886ae343309c-kube-api-access-mm4s6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:51:47.116590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:47.116562 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" Oct 30 06:51:47.116802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:47.116651 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv" event={"ID":"db4038d3-ac1b-48ec-a31e-886ae343309c","Type":"ContainerDied","Data":"eea45c6a10d920b89a1d947f7b831804db5522fc6a6945d02cda9728e9b5fbb5"} Oct 30 06:51:47.116802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:47.116693 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eea45c6a10d920b89a1d947f7b831804db5522fc6a6945d02cda9728e9b5fbb5" Oct 30 06:51:53.185337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.185281 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72"] Oct 30 06:51:53.186409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.186389 5001 cpu_manager.go:414] "containerMap: removing container" podUID="db4038d3-ac1b-48ec-a31e-886ae343309c" containerName="revoke-keys" Oct 30 06:51:53.186409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.186408 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4038d3-ac1b-48ec-a31e-886ae343309c" containerName="revoke-keys" Oct 30 06:51:53.186605 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.186594 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4038d3-ac1b-48ec-a31e-886ae343309c" containerName="revoke-keys" Oct 30 06:51:53.189685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.189665 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:53.208707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.208677 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72"] Oct 30 06:51:53.214360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.214331 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:51:53.214496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.214377 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:51:53.214496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.214412 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:51:53.215971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.215949 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:51:53.289674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.289619 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deeba46f-tqw72\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:53.289896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.289766 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deeba46f-tqw72\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:53.289986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.289965 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2zqg\" (UniqueName: \"kubernetes.io/projected/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-kube-api-access-l2zqg\") pod \"maas-api-revoke-sub-deeba46f-tqw72\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:53.391895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.391844 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2zqg\" (UniqueName: \"kubernetes.io/projected/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-kube-api-access-l2zqg\") pod \"maas-api-revoke-sub-deeba46f-tqw72\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:53.392126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.391985 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deeba46f-tqw72\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:53.392126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.392074 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deeba46f-tqw72\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:53.392798 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.392776 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deeba46f-tqw72\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:53.394705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.394681 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deeba46f-tqw72\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:53.414868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.414838 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2zqg\" (UniqueName: \"kubernetes.io/projected/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-kube-api-access-l2zqg\") pod \"maas-api-revoke-sub-deeba46f-tqw72\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:53.508449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.508362 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:53.823227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:53.823192 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72"] Oct 30 06:51:53.824427 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:51:53.824397 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755ec414_be8d_4a0b_93ba_a0a1229fbfbc.slice/crio-0d72e40f25246fa7a4cca7a83408bafe72cfa2137f9ea96df3f3f4f5510232ed WatchSource:0}: Error finding container 0d72e40f25246fa7a4cca7a83408bafe72cfa2137f9ea96df3f3f4f5510232ed: Status 404 returned error can't find the container with id 0d72e40f25246fa7a4cca7a83408bafe72cfa2137f9ea96df3f3f4f5510232ed Oct 30 06:51:54.190479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:54.190440 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" event={"ID":"755ec414-be8d-4a0b-93ba-a0a1229fbfbc","Type":"ContainerStarted","Data":"0d72e40f25246fa7a4cca7a83408bafe72cfa2137f9ea96df3f3f4f5510232ed"} Oct 30 06:51:55.201864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:55.201834 5001 generic.go:356] "Generic (PLEG): container finished" podID="755ec414-be8d-4a0b-93ba-a0a1229fbfbc" containerID="a7c024c27300d139e27a78d177a720daa5267c76858453b4712a99e1fbf64ca2" exitCode=0 Oct 30 06:51:55.202229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:55.201919 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" event={"ID":"755ec414-be8d-4a0b-93ba-a0a1229fbfbc","Type":"ContainerDied","Data":"a7c024c27300d139e27a78d177a720daa5267c76858453b4712a99e1fbf64ca2"} Oct 30 06:51:56.479267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:56.479201 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:56.543741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:56.543705 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-xks-ca\") pod \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " Oct 30 06:51:56.543978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:56.543912 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-kube-api-access-l2zqg\" (UniqueName: \"kubernetes.io/projected/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-kube-api-access-l2zqg\") pod \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " Oct 30 06:51:56.544082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:56.544020 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-openshift-ca\") pod \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\" (UID: \"755ec414-be8d-4a0b-93ba-a0a1229fbfbc\") " Oct 30 06:51:56.544450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:56.544422 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-openshift-ca" pod "755ec414-be8d-4a0b-93ba-a0a1229fbfbc" (UID: "755ec414-be8d-4a0b-93ba-a0a1229fbfbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:51:56.545013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:56.544989 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:51:56.546211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:56.546183 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-xks-ca" pod "755ec414-be8d-4a0b-93ba-a0a1229fbfbc" (UID: "755ec414-be8d-4a0b-93ba-a0a1229fbfbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:51:56.546425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:56.546404 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-kube-api-access-l2zqg" pod "755ec414-be8d-4a0b-93ba-a0a1229fbfbc" (UID: "755ec414-be8d-4a0b-93ba-a0a1229fbfbc"). InnerVolumeSpecName "kube-api-access-l2zqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:51:56.646972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:56.646936 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:51:56.646972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:56.646971 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2zqg\" (UniqueName: \"kubernetes.io/projected/755ec414-be8d-4a0b-93ba-a0a1229fbfbc-kube-api-access-l2zqg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:51:57.222945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:57.222915 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" Oct 30 06:51:57.223120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:57.222941 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72" event={"ID":"755ec414-be8d-4a0b-93ba-a0a1229fbfbc","Type":"ContainerDied","Data":"0d72e40f25246fa7a4cca7a83408bafe72cfa2137f9ea96df3f3f4f5510232ed"} Oct 30 06:51:57.223120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:57.222991 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d72e40f25246fa7a4cca7a83408bafe72cfa2137f9ea96df3f3f4f5510232ed" Oct 30 06:51:58.061445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:58.061408 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9"] Oct 30 06:51:58.076618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:58.076582 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96617b2b-l9bw9"] Oct 30 06:51:58.121036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:51:58.121004 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda" path="/var/lib/kubelet/pods/500b90ce-a7a2-4aeb-bdeb-65f4eb79ecda/volumes" Oct 30 06:52:04.868238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:04.868200 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq"] Oct 30 06:52:05.808520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:05.808481 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn"] Oct 30 06:52:08.266196 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:52:08.266153 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:52:09.869256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:09.869209 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" podUID="f5969015-c270-4671-8af3-d5a565fa48a3" containerName="payload-processing" containerID="cri-o://03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04" gracePeriod=25 Oct 30 06:52:10.181907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.181881 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:52:10.272455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.272409 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-plugins-config-volume\") pod \"f5969015-c270-4671-8af3-d5a565fa48a3\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " Oct 30 06:52:10.272644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.272556 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-provider-credentials\") pod \"f5969015-c270-4671-8af3-d5a565fa48a3\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " Oct 30 06:52:10.272750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.272693 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-routing-overlay-volume\") pod \"f5969015-c270-4671-8af3-d5a565fa48a3\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " Oct 30 06:52:10.272888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.272870 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-kube-api-access-ll9pj\" (UniqueName: \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-kube-api-access-ll9pj\") pod \"f5969015-c270-4671-8af3-d5a565fa48a3\" (UID: \"f5969015-c270-4671-8af3-d5a565fa48a3\") " Oct 30 06:52:10.273048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.273014 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-plugins-config-volume" pod "f5969015-c270-4671-8af3-d5a565fa48a3" (UID: "f5969015-c270-4671-8af3-d5a565fa48a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:52:10.273195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.273060 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-routing-overlay-volume" pod "f5969015-c270-4671-8af3-d5a565fa48a3" (UID: "f5969015-c270-4671-8af3-d5a565fa48a3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:52:10.274521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.274472 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:10.274521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.274499 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5969015-c270-4671-8af3-d5a565fa48a3-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:10.276090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.276056 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-kube-api-access-ll9pj" pod "f5969015-c270-4671-8af3-d5a565fa48a3" (UID: "f5969015-c270-4671-8af3-d5a565fa48a3"). InnerVolumeSpecName "kube-api-access-ll9pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:52:10.276196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.276103 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-provider-credentials" pod "f5969015-c270-4671-8af3-d5a565fa48a3" (UID: "f5969015-c270-4671-8af3-d5a565fa48a3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:52:10.376057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.376023 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:10.376057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.376054 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll9pj\" (UniqueName: \"kubernetes.io/projected/f5969015-c270-4671-8af3-d5a565fa48a3-kube-api-access-ll9pj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:10.443418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.443332 5001 generic.go:356] "Generic (PLEG): container finished" podID="f5969015-c270-4671-8af3-d5a565fa48a3" containerID="03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04" exitCode=0 Oct 30 06:52:10.443418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.443371 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" Oct 30 06:52:10.443418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.443409 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" event={"ID":"f5969015-c270-4671-8af3-d5a565fa48a3","Type":"ContainerDied","Data":"03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04"} Oct 30 06:52:10.443678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.443466 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq" event={"ID":"f5969015-c270-4671-8af3-d5a565fa48a3","Type":"ContainerDied","Data":"730393e3899f9865bbe802d0ba5560aa0eeb75adbbe843d0f82b10440fa616ec"} Oct 30 06:52:10.443678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.443497 5001 scope.go:122] "RemoveContainer" containerID="03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04" Oct 30 06:52:10.456641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.456620 5001 scope.go:122] "RemoveContainer" containerID="03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04" Oct 30 06:52:10.456951 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:52:10.456930 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04\": container with ID starting with 03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04 not found: ID does not exist" containerID="03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04" Oct 30 06:52:10.457000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.456960 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04"} err="failed to get container status \"03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04\": rpc error: code = NotFound desc = could not find container \"03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04\": container with ID starting with 03dcb1ba41fb74829e3a7d6d11ffe36b934f73a94e247888b41d70df4bb2da04 not found: ID does not exist" Oct 30 06:52:10.497933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.497895 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq"] Oct 30 06:52:10.503418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.503390 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-69d6wx4vq"] Oct 30 06:52:10.809032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:10.808934 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" podUID="e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf" containerName="payload-processing" containerID="cri-o://3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01" gracePeriod=25 Oct 30 06:52:11.124371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.124299 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:52:11.192624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.192590 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-kube-api-access-gd85l\" (UniqueName: \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-kube-api-access-gd85l\") pod \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " Oct 30 06:52:11.192850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.192834 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-provider-credentials\") pod \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " Oct 30 06:52:11.193028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.193009 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-plugins-config-volume\") pod \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " Oct 30 06:52:11.193142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.193091 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-routing-overlay-volume\") pod \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\" (UID: \"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf\") " Oct 30 06:52:11.194260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.193470 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-plugins-config-volume" pod "e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf" (UID: "e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:52:11.194260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.193657 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-routing-overlay-volume" pod "e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf" (UID: "e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:52:11.194260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.194019 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:11.194260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.194042 5001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-routing-overlay-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:11.195730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.195703 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-provider-credentials" pod "e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf" (UID: "e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:52:11.195904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.195873 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-kube-api-access-gd85l" pod "e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf" (UID: "e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf"). InnerVolumeSpecName "kube-api-access-gd85l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:52:11.295720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.295688 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd85l\" (UniqueName: \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-kube-api-access-gd85l\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:11.295720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.295719 5001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf-provider-credentials\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:11.453597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.453562 5001 generic.go:356] "Generic (PLEG): container finished" podID="e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf" containerID="3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01" exitCode=0 Oct 30 06:52:11.453788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.453619 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" Oct 30 06:52:11.453788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.453652 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" event={"ID":"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf","Type":"ContainerDied","Data":"3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01"} Oct 30 06:52:11.453788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.453703 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn" event={"ID":"e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf","Type":"ContainerDied","Data":"e859889359eb21f0152edc4d2e1d9a93c68e61c37cc406780c74623561944e4e"} Oct 30 06:52:11.453788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.453734 5001 scope.go:122] "RemoveContainer" containerID="3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01" Oct 30 06:52:11.473217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.473199 5001 scope.go:122] "RemoveContainer" containerID="3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01" Oct 30 06:52:11.473557 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:52:11.473534 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01\": container with ID starting with 3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01 not found: ID does not exist" containerID="3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01" Oct 30 06:52:11.473645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.473568 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01"} err="failed to get container status \"3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01\": rpc error: code = NotFound desc = could not find container \"3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01\": container with ID starting with 3c1aa198bb1b9e9a225d2c375c9f1b72f7b229a2233e2f61c10d6f3c7e383c01 not found: ID does not exist" Oct 30 06:52:11.510232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.510197 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn"] Oct 30 06:52:11.521378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.521351 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a0083e/payload-processing-external-model-e2e-extmod-b-a0083e-5664rvgvn"] Oct 30 06:52:11.810540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:11.810452 5001 scope.go:122] "RemoveContainer" containerID="4ed62793a8545d232f7b743335f8f42ef5f4f513fbe5883eae805546aa1ea9c7" Oct 30 06:52:12.121898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.121854 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf" path="/var/lib/kubelet/pods/e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf/volumes" Oct 30 06:52:12.122494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.122469 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5969015-c270-4671-8af3-d5a565fa48a3" path="/var/lib/kubelet/pods/f5969015-c270-4671-8af3-d5a565fa48a3/volumes" Oct 30 06:52:12.825566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.825530 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f"] Oct 30 06:52:12.826464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.826446 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf" containerName="payload-processing" Oct 30 06:52:12.826521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.826465 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf" containerName="payload-processing" Oct 30 06:52:12.826521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.826501 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f5969015-c270-4671-8af3-d5a565fa48a3" containerName="payload-processing" Oct 30 06:52:12.826521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.826507 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5969015-c270-4671-8af3-d5a565fa48a3" containerName="payload-processing" Oct 30 06:52:12.826627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.826525 5001 cpu_manager.go:414] "containerMap: removing container" podUID="755ec414-be8d-4a0b-93ba-a0a1229fbfbc" containerName="revoke-keys" Oct 30 06:52:12.826627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.826531 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755ec414-be8d-4a0b-93ba-a0a1229fbfbc" containerName="revoke-keys" Oct 30 06:52:12.826690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.826661 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="755ec414-be8d-4a0b-93ba-a0a1229fbfbc" containerName="revoke-keys" Oct 30 06:52:12.826690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.826677 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c7cf6b-7f60-4d47-8ebf-f6ada29eaacf" containerName="payload-processing" Oct 30 06:52:12.826690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.826687 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5969015-c270-4671-8af3-d5a565fa48a3" containerName="payload-processing" Oct 30 06:52:12.832439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.832416 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:12.853704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.853675 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f"] Oct 30 06:52:12.858661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.858624 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:52:12.858661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.858630 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:52:12.858872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.858667 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:52:12.866417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.866383 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:52:12.922378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.922336 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:12.922559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.922451 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27svb\" (UniqueName: \"kubernetes.io/projected/651a7ffa-5b7e-4e12-bc5a-772013821383-kube-api-access-27svb\") pod \"maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:12.922623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:12.922609 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:13.024690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:13.024652 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:13.024690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:13.024707 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27svb\" (UniqueName: \"kubernetes.io/projected/651a7ffa-5b7e-4e12-bc5a-772013821383-kube-api-access-27svb\") pod \"maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:13.024986 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:13.024799 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:13.025379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:13.025356 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:13.027464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:13.027430 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:13.048678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:13.048640 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27svb\" (UniqueName: \"kubernetes.io/projected/651a7ffa-5b7e-4e12-bc5a-772013821383-kube-api-access-27svb\") pod \"maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:13.147041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:13.147007 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:13.459539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:13.459508 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f"] Oct 30 06:52:13.461951 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:52:13.461919 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651a7ffa_5b7e_4e12_bc5a_772013821383.slice/crio-fd2df8ef02f7e76b03b50b1bbf9b578c53005dffe1ca2cd141896e39242d4ef0 WatchSource:0}: Error finding container fd2df8ef02f7e76b03b50b1bbf9b578c53005dffe1ca2cd141896e39242d4ef0: Status 404 returned error can't find the container with id fd2df8ef02f7e76b03b50b1bbf9b578c53005dffe1ca2cd141896e39242d4ef0 Oct 30 06:52:13.476765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:13.476735 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" event={"ID":"651a7ffa-5b7e-4e12-bc5a-772013821383","Type":"ContainerStarted","Data":"fd2df8ef02f7e76b03b50b1bbf9b578c53005dffe1ca2cd141896e39242d4ef0"} Oct 30 06:52:14.564175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:14.564136 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" event={"ID":"651a7ffa-5b7e-4e12-bc5a-772013821383","Type":"ContainerStarted","Data":"a91ebf922da226cc273e56be97d3512a9a2ccb89891cf1ea50ff188a3d6d487e"} Oct 30 06:52:14.604743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:14.604685 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" podStartSLOduration=2.604665373 podStartE2EDuration="2.604665373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:52:14.604459267 +0000 UTC m=+3548.600416278" watchObservedRunningTime="2026-10-30 06:52:14.604665373 +0000 UTC m=+3548.600622386" Oct 30 06:52:15.045106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:15.045065 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb"] Oct 30 06:52:15.078296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:15.078254 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87632f97-28gdb"] Oct 30 06:52:15.580033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:15.579995 5001 generic.go:356] "Generic (PLEG): container finished" podID="651a7ffa-5b7e-4e12-bc5a-772013821383" containerID="a91ebf922da226cc273e56be97d3512a9a2ccb89891cf1ea50ff188a3d6d487e" exitCode=0 Oct 30 06:52:15.580555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:15.580076 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" event={"ID":"651a7ffa-5b7e-4e12-bc5a-772013821383","Type":"ContainerDied","Data":"a91ebf922da226cc273e56be97d3512a9a2ccb89891cf1ea50ff188a3d6d487e"} Oct 30 06:52:16.121388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:16.121354 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66" path="/var/lib/kubelet/pods/4680fa3d-ad37-4552-bd5c-ea3ee7a2ad66/volumes" Oct 30 06:52:16.789323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:16.789282 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:16.902912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:16.902872 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-openshift-ca\") pod \"651a7ffa-5b7e-4e12-bc5a-772013821383\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " Oct 30 06:52:16.903118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:16.903032 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-xks-ca\") pod \"651a7ffa-5b7e-4e12-bc5a-772013821383\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " Oct 30 06:52:16.903118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:16.903083 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651a7ffa-5b7e-4e12-bc5a-772013821383-kube-api-access-27svb\" (UniqueName: \"kubernetes.io/projected/651a7ffa-5b7e-4e12-bc5a-772013821383-kube-api-access-27svb\") pod \"651a7ffa-5b7e-4e12-bc5a-772013821383\" (UID: \"651a7ffa-5b7e-4e12-bc5a-772013821383\") " Oct 30 06:52:16.903377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:16.903349 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-openshift-ca" pod "651a7ffa-5b7e-4e12-bc5a-772013821383" (UID: "651a7ffa-5b7e-4e12-bc5a-772013821383"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:52:16.903951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:16.903923 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:16.906221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:16.906194 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-xks-ca" pod "651a7ffa-5b7e-4e12-bc5a-772013821383" (UID: "651a7ffa-5b7e-4e12-bc5a-772013821383"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:52:16.906344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:16.906221 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651a7ffa-5b7e-4e12-bc5a-772013821383-kube-api-access-27svb" pod "651a7ffa-5b7e-4e12-bc5a-772013821383" (UID: "651a7ffa-5b7e-4e12-bc5a-772013821383"). InnerVolumeSpecName "kube-api-access-27svb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:52:17.005696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:17.005653 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651a7ffa-5b7e-4e12-bc5a-772013821383-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:17.005696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:17.005686 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27svb\" (UniqueName: \"kubernetes.io/projected/651a7ffa-5b7e-4e12-bc5a-772013821383-kube-api-access-27svb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:17.604758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:17.604723 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" Oct 30 06:52:17.604940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:17.604761 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f" event={"ID":"651a7ffa-5b7e-4e12-bc5a-772013821383","Type":"ContainerDied","Data":"fd2df8ef02f7e76b03b50b1bbf9b578c53005dffe1ca2cd141896e39242d4ef0"} Oct 30 06:52:17.604940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:17.604851 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd2df8ef02f7e76b03b50b1bbf9b578c53005dffe1ca2cd141896e39242d4ef0" Oct 30 06:52:23.337874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:23.337607 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v"] Oct 30 06:52:26.035653 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:26.035615 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4"] Oct 30 06:52:26.048675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:26.048625 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0c70f51-lmpr4"] Oct 30 06:52:26.120107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:26.120074 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b8e92e-ca86-4a1d-ada3-f99d7f5de94d" path="/var/lib/kubelet/pods/84b8e92e-ca86-4a1d-ada3-f99d7f5de94d/volumes" Oct 30 06:52:27.926958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:27.926922 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:52:28.338149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:28.338048 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" podUID="3eb1ebd4-bede-4c40-9e3e-5f689e995b9e" containerName="payload-processing" containerID="cri-o://577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6" gracePeriod=25 Oct 30 06:52:32.975705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:32.975667 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" podUID="3eb1ebd4-bede-4c40-9e3e-5f689e995b9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:52:37.450981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.450947 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt"] Oct 30 06:52:37.479328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.475443 5001 cpu_manager.go:414] "containerMap: removing container" podUID="651a7ffa-5b7e-4e12-bc5a-772013821383" containerName="revoke-keys" Oct 30 06:52:37.479328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.475473 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651a7ffa-5b7e-4e12-bc5a-772013821383" containerName="revoke-keys" Oct 30 06:52:37.479328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.475843 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="651a7ffa-5b7e-4e12-bc5a-772013821383" containerName="revoke-keys" Oct 30 06:52:37.493073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.493048 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.499690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.499651 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt"] Oct 30 06:52:37.508032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.508004 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8baf7e\"/\"kube-root-ca.crt\"" Oct 30 06:52:37.512658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.512628 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8baf7e\"/\"openshift-service-ca.crt\"" Oct 30 06:52:37.524409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.524375 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8baf7e\"/\"default-dockercfg-87zzw\"" Oct 30 06:52:37.524567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.524479 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8baf7e\"/\"test-model-8baf7e-kserve-self-signed-certs\"" Oct 30 06:52:37.550781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.550751 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6483839b-0d24-4763-b6c8-10337a352e0a-tls-certs\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.550962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.550810 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hgpf\" (UniqueName: \"kubernetes.io/projected/6483839b-0d24-4763-b6c8-10337a352e0a-kube-api-access-7hgpf\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.551033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.550961 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-model-cache\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.551033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.551008 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-tmp-dir\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.551152 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.551101 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-home\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.551276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.551260 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-dshm\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.674332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.657512 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-model-cache\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.678192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.678162 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-tmp-dir\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.678585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.678562 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-home\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.683401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.679120 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-dshm\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.683911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.683878 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6483839b-0d24-4763-b6c8-10337a352e0a-tls-certs\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.684082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.684065 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hgpf\" (UniqueName: \"kubernetes.io/projected/6483839b-0d24-4763-b6c8-10337a352e0a-kube-api-access-7hgpf\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.685836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.682429 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-home\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.685836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.681660 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-tmp-dir\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.685836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.675015 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-model-cache\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.687323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.687285 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-dshm\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.729464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.720510 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hgpf\" (UniqueName: \"kubernetes.io/projected/6483839b-0d24-4763-b6c8-10337a352e0a-kube-api-access-7hgpf\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.746365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.733697 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6483839b-0d24-4763-b6c8-10337a352e0a-tls-certs\") pod \"test-model-8baf7e-kserve-676bf47589-pjmwt\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:37.818167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:37.818128 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:38.294265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:38.294231 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt"] Oct 30 06:52:38.294816 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:52:38.294757 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6483839b_0d24_4763_b6c8_10337a352e0a.slice/crio-23f0cdad2da2b9e1625b1bd59bd35d8c3aa58bc7d7eabf73770775910e92d76e WatchSource:0}: Error finding container 23f0cdad2da2b9e1625b1bd59bd35d8c3aa58bc7d7eabf73770775910e92d76e: Status 404 returned error can't find the container with id 23f0cdad2da2b9e1625b1bd59bd35d8c3aa58bc7d7eabf73770775910e92d76e Oct 30 06:52:38.306739 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:52:38.306706 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:52:38.891354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:38.891296 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" event={"ID":"6483839b-0d24-4763-b6c8-10337a352e0a","Type":"ContainerStarted","Data":"4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8"} Oct 30 06:52:38.891872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:38.891372 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" event={"ID":"6483839b-0d24-4763-b6c8-10337a352e0a","Type":"ContainerStarted","Data":"23f0cdad2da2b9e1625b1bd59bd35d8c3aa58bc7d7eabf73770775910e92d76e"} Oct 30 06:52:38.936898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:38.936830 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" podStartSLOduration=1.9368095950000002 podStartE2EDuration="1.936809595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:52:38.933426314 +0000 UTC m=+3572.929383336" watchObservedRunningTime="2026-10-30 06:52:38.936809595 +0000 UTC m=+3572.932766610" Oct 30 06:52:39.819064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:39.819029 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:42.974915 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:42.974878 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" podUID="3eb1ebd4-bede-4c40-9e3e-5f689e995b9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:52:45.961650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:45.961618 5001 generic.go:356] "Generic (PLEG): container finished" podID="ba380ad4-2195-4cfc-b197-d1568adc5140" containerID="1a5cfba7280c2a840b5c88950a1637330af9d5e2aff650eaf6be6631c180a391" exitCode=137 Oct 30 06:52:45.976095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:45.961697 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" event={"ID":"ba380ad4-2195-4cfc-b197-d1568adc5140","Type":"ContainerDied","Data":"1a5cfba7280c2a840b5c88950a1637330af9d5e2aff650eaf6be6631c180a391"} Oct 30 06:52:45.976095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:45.962183 5001 scope.go:122] "RemoveContainer" containerID="1a5cfba7280c2a840b5c88950a1637330af9d5e2aff650eaf6be6631c180a391" Oct 30 06:52:46.972506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:46.972469 5001 generic.go:356] "Generic (PLEG): container finished" podID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerID="5a2371cec333f16b59f9054c7132a50bd8827237c999ae5f964b7d337f57a303" exitCode=0 Oct 30 06:52:46.973067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:46.972559 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" event={"ID":"cd760fe2-e8c7-42f9-8a3f-40ec810159b4","Type":"ContainerDied","Data":"5a2371cec333f16b59f9054c7132a50bd8827237c999ae5f964b7d337f57a303"} Oct 30 06:52:46.973067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:46.973015 5001 scope.go:122] "RemoveContainer" containerID="5a2371cec333f16b59f9054c7132a50bd8827237c999ae5f964b7d337f57a303" Oct 30 06:52:46.975040 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:46.975018 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" event={"ID":"ba380ad4-2195-4cfc-b197-d1568adc5140","Type":"ContainerStarted","Data":"2ec3b96a4da9b01e817dbee5ab545c15c3e1bf17f2736582d0dddebcf5c302b7"} Oct 30 06:52:46.975162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:46.975063 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:52:46.975162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:46.975134 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:52:46.975264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:46.975174 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:52:46.975264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:46.975207 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:52:47.819192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:47.819151 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:47.832707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:47.832674 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:47.991005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:47.990968 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" event={"ID":"cd760fe2-e8c7-42f9-8a3f-40ec810159b4","Type":"ContainerStarted","Data":"58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160"} Oct 30 06:52:48.002419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:48.002391 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:52:48.691353 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:48.691280 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:52:48.793277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:48.793224 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-plugins-config-volume\") pod \"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e\" (UID: \"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e\") " Oct 30 06:52:48.793503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:48.793455 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-kube-api-access-9r96l\" (UniqueName: \"kubernetes.io/projected/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-kube-api-access-9r96l\") pod \"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e\" (UID: \"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e\") " Oct 30 06:52:48.793680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:48.793652 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-plugins-config-volume" pod "3eb1ebd4-bede-4c40-9e3e-5f689e995b9e" (UID: "3eb1ebd4-bede-4c40-9e3e-5f689e995b9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:52:48.794224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:48.794197 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:48.797387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:48.795852 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-kube-api-access-9r96l" pod "3eb1ebd4-bede-4c40-9e3e-5f689e995b9e" (UID: "3eb1ebd4-bede-4c40-9e3e-5f689e995b9e"). InnerVolumeSpecName "kube-api-access-9r96l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:52:48.895875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:48.895840 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r96l\" (UniqueName: \"kubernetes.io/projected/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e-kube-api-access-9r96l\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:48.981129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:48.981050 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:52:48.987102 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:48.987077 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:52:49.004969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:49.004939 5001 generic.go:356] "Generic (PLEG): container finished" podID="3eb1ebd4-bede-4c40-9e3e-5f689e995b9e" containerID="577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6" exitCode=0 Oct 30 06:52:49.005471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:49.004977 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" Oct 30 06:52:49.005471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:49.005016 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" event={"ID":"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e","Type":"ContainerDied","Data":"577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6"} Oct 30 06:52:49.005471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:49.005075 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v" event={"ID":"3eb1ebd4-bede-4c40-9e3e-5f689e995b9e","Type":"ContainerDied","Data":"2a5daf115dc0d04e3d1c730456ff987aaae39fe5a6f3fc60012dcd4fdba7a0e3"} Oct 30 06:52:49.005471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:49.005102 5001 scope.go:122] "RemoveContainer" containerID="577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6" Oct 30 06:52:49.005471 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:49.005429 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:52:49.006524 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:49.006506 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:52:49.018075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:49.018058 5001 scope.go:122] "RemoveContainer" containerID="577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6" Oct 30 06:52:49.018406 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:52:49.018385 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6\": container with ID starting with 577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6 not found: ID does not exist" containerID="577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6" Oct 30 06:52:49.018473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:49.018412 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6"} err="failed to get container status \"577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6\": rpc error: code = NotFound desc = could not find container \"577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6\": container with ID starting with 577f30182f3a67ee2518099d63045561fd042045e0189207a1374fb298308fe6 not found: ID does not exist" Oct 30 06:52:49.125412 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:49.125379 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v"] Oct 30 06:52:49.131445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:49.131416 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d2923-6d6d74576d-l294v"] Oct 30 06:52:50.120323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:50.120280 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb1ebd4-bede-4c40-9e3e-5f689e995b9e" path="/var/lib/kubelet/pods/3eb1ebd4-bede-4c40-9e3e-5f689e995b9e/volumes" Oct 30 06:52:55.257126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.257089 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h"] Oct 30 06:52:55.258065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.258047 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3eb1ebd4-bede-4c40-9e3e-5f689e995b9e" containerName="payload-processing" Oct 30 06:52:55.258111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.258067 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb1ebd4-bede-4c40-9e3e-5f689e995b9e" containerName="payload-processing" Oct 30 06:52:55.258253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.258243 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb1ebd4-bede-4c40-9e3e-5f689e995b9e" containerName="payload-processing" Oct 30 06:52:55.262159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.262141 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:55.283984 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.283954 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h"] Oct 30 06:52:55.287806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.287780 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:52:55.288579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.288561 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:52:55.288579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.288563 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:52:55.288741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.288582 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:52:55.423589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.423551 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805e52b6-dvl7h\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:55.423790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.423618 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805e52b6-dvl7h\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:55.423842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.423798 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvwfz\" (UniqueName: \"kubernetes.io/projected/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-kube-api-access-jvwfz\") pod \"maas-api-revoke-sub-805e52b6-dvl7h\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:55.525667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.525567 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805e52b6-dvl7h\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:55.525667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.525650 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805e52b6-dvl7h\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:55.525913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.525787 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvwfz\" (UniqueName: \"kubernetes.io/projected/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-kube-api-access-jvwfz\") pod \"maas-api-revoke-sub-805e52b6-dvl7h\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:55.526512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.526489 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805e52b6-dvl7h\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:55.528236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.528212 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805e52b6-dvl7h\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:55.548608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.548571 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvwfz\" (UniqueName: \"kubernetes.io/projected/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-kube-api-access-jvwfz\") pod \"maas-api-revoke-sub-805e52b6-dvl7h\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:55.576414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.576383 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:55.889746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:55.889716 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h"] Oct 30 06:52:55.890643 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:52:55.890617 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ddf0da_b0bf_4c5e_9a96_7dc402a97d5f.slice/crio-4c73feaf02204202729cb86d3a198d08f43e7880b31ebb36a94978b04c8bb03e WatchSource:0}: Error finding container 4c73feaf02204202729cb86d3a198d08f43e7880b31ebb36a94978b04c8bb03e: Status 404 returned error can't find the container with id 4c73feaf02204202729cb86d3a198d08f43e7880b31ebb36a94978b04c8bb03e Oct 30 06:52:56.074411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:56.074370 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" event={"ID":"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f","Type":"ContainerStarted","Data":"4c73feaf02204202729cb86d3a198d08f43e7880b31ebb36a94978b04c8bb03e"} Oct 30 06:52:57.085492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:57.085456 5001 generic.go:356] "Generic (PLEG): container finished" podID="47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f" containerID="04e718a56ca683afa534828030465ec6e453adf36c28b9bc5da36e98e4910953" exitCode=0 Oct 30 06:52:57.085932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:57.085548 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" event={"ID":"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f","Type":"ContainerDied","Data":"04e718a56ca683afa534828030465ec6e453adf36c28b9bc5da36e98e4910953"} Oct 30 06:52:57.993992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:57.993962 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hbpn7" Oct 30 06:52:58.313406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:58.313380 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:58.490769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:58.490737 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-kube-api-access-jvwfz\" (UniqueName: \"kubernetes.io/projected/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-kube-api-access-jvwfz\") pod \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " Oct 30 06:52:58.491092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:58.491077 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-openshift-ca\") pod \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " Oct 30 06:52:58.491140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:58.491112 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-xks-ca\") pod \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\" (UID: \"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f\") " Oct 30 06:52:58.491455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:58.491424 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-openshift-ca" pod "47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f" (UID: "47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:52:58.491654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:58.491630 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:58.493098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:58.493078 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-kube-api-access-jvwfz" pod "47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f" (UID: "47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f"). InnerVolumeSpecName "kube-api-access-jvwfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:52:58.493288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:58.493271 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-xks-ca" pod "47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f" (UID: "47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:52:58.593226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:58.593146 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:58.593226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:58.593175 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvwfz\" (UniqueName: \"kubernetes.io/projected/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f-kube-api-access-jvwfz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:52:59.110442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:59.110405 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" Oct 30 06:52:59.110625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:59.110460 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h" event={"ID":"47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f","Type":"ContainerDied","Data":"4c73feaf02204202729cb86d3a198d08f43e7880b31ebb36a94978b04c8bb03e"} Oct 30 06:52:59.110625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:52:59.110507 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c73feaf02204202729cb86d3a198d08f43e7880b31ebb36a94978b04c8bb03e" Oct 30 06:53:03.797335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.792579 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f"] Oct 30 06:53:03.821343 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.820853 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2d2923-f5x4f"] Oct 30 06:53:03.909886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.909848 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs"] Oct 30 06:53:03.911033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.911004 5001 cpu_manager.go:414] "containerMap: removing container" podUID="47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f" containerName="revoke-keys" Oct 30 06:53:03.911033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.911030 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f" containerName="revoke-keys" Oct 30 06:53:03.911346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.911330 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f" containerName="revoke-keys" Oct 30 06:53:03.915270 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.915250 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:03.934644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.934611 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-be48b4\"/\"openshift-service-ca.crt\"" Oct 30 06:53:03.940669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.940636 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs"] Oct 30 06:53:03.943458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.943425 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-be48b4\"/\"default-dockercfg-fmtjl\"" Oct 30 06:53:03.943458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.943425 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-be48b4\"/\"kube-root-ca.crt\"" Oct 30 06:53:03.943662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:03.943475 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-be48b4\"/\"test-model-be48b4-kserve-self-signed-certs\"" Oct 30 06:53:04.027481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.027444 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tls-certs\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.027720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.027591 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-dshm\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.027720 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.027693 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjdx9\" (UniqueName: \"kubernetes.io/projected/f54e188f-35ed-4d3a-b66a-9ed12b084c36-kube-api-access-sjdx9\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.027861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.027838 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-model-cache\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.027994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.027964 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-home\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.028155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.028137 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tmp-dir\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.121481 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.121442 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651a7ffa-5b7e-4e12-bc5a-772013821383" path="/var/lib/kubelet/pods/651a7ffa-5b7e-4e12-bc5a-772013821383/volumes" Oct 30 06:53:04.129522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.129496 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-model-cache\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.129635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.129555 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-home\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.129687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.129658 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tmp-dir\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.129736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.129711 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tls-certs\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.129806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.129781 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-dshm\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.129882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.129861 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-model-cache\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.129945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.129916 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjdx9\" (UniqueName: \"kubernetes.io/projected/f54e188f-35ed-4d3a-b66a-9ed12b084c36-kube-api-access-sjdx9\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.129945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.129934 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-home\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.130074 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.130043 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tmp-dir\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.132171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.132145 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-dshm\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.132430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.132415 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tls-certs\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.175084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.175050 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjdx9\" (UniqueName: \"kubernetes.io/projected/f54e188f-35ed-4d3a-b66a-9ed12b084c36-kube-api-access-sjdx9\") pod \"test-model-be48b4-kserve-855dffb897-jcnfs\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.259327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.248092 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:04.665739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:04.665181 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs"] Oct 30 06:53:05.270407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:05.270365 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" event={"ID":"f54e188f-35ed-4d3a-b66a-9ed12b084c36","Type":"ContainerStarted","Data":"d57154351f35b300dd76fdfa5283289588d6de04e500ad002d27e327a6cc8d5d"} Oct 30 06:53:05.270407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:05.270415 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" event={"ID":"f54e188f-35ed-4d3a-b66a-9ed12b084c36","Type":"ContainerStarted","Data":"4643279df88b2d9d1e6df1012c380b7f508421a2edc4f3ba8e03a395269890d7"} Oct 30 06:53:05.312067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:05.312010 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" podStartSLOduration=2.311996598 podStartE2EDuration="2.311996598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:53:05.310553878 +0000 UTC m=+3599.306510892" watchObservedRunningTime="2026-10-30 06:53:05.311996598 +0000 UTC m=+3599.307953607" Oct 30 06:53:06.254216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:06.254182 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:07.136673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:07.136642 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:53:07.151449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:07.151398 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:53:08.269174 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:53:08.269141 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:53:08.500930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.500895 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn"] Oct 30 06:53:08.506118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.506098 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:08.526429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.526358 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn"] Oct 30 06:53:08.530148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.530120 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:53:08.530277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.530169 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:53:08.532419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.532387 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:53:08.532537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.532520 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:53:08.640782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.640745 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6shm\" (UniqueName: \"kubernetes.io/projected/2bec1809-a925-451b-9d14-7bdf5377bfb3-kube-api-access-b6shm\") pod \"maas-api-revoke-sub-9da4cdf2-mqlkn\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:08.640962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.640873 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9da4cdf2-mqlkn\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:08.640962 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.640944 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9da4cdf2-mqlkn\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:08.742983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.742945 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9da4cdf2-mqlkn\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:08.743180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.743166 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6shm\" (UniqueName: \"kubernetes.io/projected/2bec1809-a925-451b-9d14-7bdf5377bfb3-kube-api-access-b6shm\") pod \"maas-api-revoke-sub-9da4cdf2-mqlkn\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:08.743259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.743222 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9da4cdf2-mqlkn\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:08.743889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.743858 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9da4cdf2-mqlkn\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:08.745765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.745739 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9da4cdf2-mqlkn\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:08.766100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.766075 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6shm\" (UniqueName: \"kubernetes.io/projected/2bec1809-a925-451b-9d14-7bdf5377bfb3-kube-api-access-b6shm\") pod \"maas-api-revoke-sub-9da4cdf2-mqlkn\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:08.826171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:08.826084 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:09.141699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:09.141664 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn"] Oct 30 06:53:09.142819 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:53:09.142788 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bec1809_a925_451b_9d14_7bdf5377bfb3.slice/crio-4625e8cb222f05485f2485fd951d734df39ccb9b97afb2d0d2150f21a9a534ca WatchSource:0}: Error finding container 4625e8cb222f05485f2485fd951d734df39ccb9b97afb2d0d2150f21a9a534ca: Status 404 returned error can't find the container with id 4625e8cb222f05485f2485fd951d734df39ccb9b97afb2d0d2150f21a9a534ca Oct 30 06:53:09.324205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:09.324162 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" event={"ID":"2bec1809-a925-451b-9d14-7bdf5377bfb3","Type":"ContainerStarted","Data":"4625e8cb222f05485f2485fd951d734df39ccb9b97afb2d0d2150f21a9a534ca"} Oct 30 06:53:10.039428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:10.039392 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp"] Oct 30 06:53:10.049300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:10.049266 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9"] Oct 30 06:53:10.054410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:10.054381 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad1f1ff-jrnwp"] Oct 30 06:53:10.060170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:10.060144 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1af0fc-9lph9"] Oct 30 06:53:10.122503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:10.122465 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8989eb-ab37-431b-b217-bed1486e939d" path="/var/lib/kubelet/pods/3c8989eb-ab37-431b-b217-bed1486e939d/volumes" Oct 30 06:53:10.123204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:10.123183 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99bdccb2-468d-4cfe-9687-f37ad6391739" path="/var/lib/kubelet/pods/99bdccb2-468d-4cfe-9687-f37ad6391739/volumes" Oct 30 06:53:10.335898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:10.335795 5001 generic.go:356] "Generic (PLEG): container finished" podID="2bec1809-a925-451b-9d14-7bdf5377bfb3" containerID="09616e978afac430d6b02a74a1412b71a766f5ebf2a412c6043cf8b99261f195" exitCode=0 Oct 30 06:53:10.336410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:10.335890 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" event={"ID":"2bec1809-a925-451b-9d14-7bdf5377bfb3","Type":"ContainerDied","Data":"09616e978afac430d6b02a74a1412b71a766f5ebf2a412c6043cf8b99261f195"} Oct 30 06:53:11.218828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.218788 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6"] Oct 30 06:53:11.219060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.219032 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" containerID="cri-o://f6080c7fb47873b68291a97b3faf495e9b3302bc6ea7a31767abc7392bc5af79" gracePeriod=30 Oct 30 06:53:11.773767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.773736 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:11.848287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.848240 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-xks-ca\") pod \"2bec1809-a925-451b-9d14-7bdf5377bfb3\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " Oct 30 06:53:11.848532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.848436 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-openshift-ca\") pod \"2bec1809-a925-451b-9d14-7bdf5377bfb3\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " Oct 30 06:53:11.848773 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.848750 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bec1809-a925-451b-9d14-7bdf5377bfb3-kube-api-access-b6shm\" (UniqueName: \"kubernetes.io/projected/2bec1809-a925-451b-9d14-7bdf5377bfb3-kube-api-access-b6shm\") pod \"2bec1809-a925-451b-9d14-7bdf5377bfb3\" (UID: \"2bec1809-a925-451b-9d14-7bdf5377bfb3\") " Oct 30 06:53:11.855523 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.855483 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-openshift-ca" pod "2bec1809-a925-451b-9d14-7bdf5377bfb3" (UID: "2bec1809-a925-451b-9d14-7bdf5377bfb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:11.859547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.859511 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5"] Oct 30 06:53:11.861254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.861225 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2bec1809-a925-451b-9d14-7bdf5377bfb3" containerName="revoke-keys" Oct 30 06:53:11.861254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.861255 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bec1809-a925-451b-9d14-7bdf5377bfb3" containerName="revoke-keys" Oct 30 06:53:11.861561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.861540 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bec1809-a925-451b-9d14-7bdf5377bfb3" containerName="revoke-keys" Oct 30 06:53:11.863091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.862524 5001 scope.go:122] "RemoveContainer" containerID="3a73613f87b620391c6ac4122ea2f12af3f026a1671a16b6f01eeb018dccf5f2" Oct 30 06:53:11.863916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.863871 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-xks-ca" pod "2bec1809-a925-451b-9d14-7bdf5377bfb3" (UID: "2bec1809-a925-451b-9d14-7bdf5377bfb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:11.865666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.865634 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bec1809-a925-451b-9d14-7bdf5377bfb3-kube-api-access-b6shm" pod "2bec1809-a925-451b-9d14-7bdf5377bfb3" (UID: "2bec1809-a925-451b-9d14-7bdf5377bfb3"). InnerVolumeSpecName "kube-api-access-b6shm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:11.867579 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.867559 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:11.885355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.881784 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5"] Oct 30 06:53:11.951600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.951561 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:11.951853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.951831 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:11.952091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.952061 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5grq\" (UniqueName: \"kubernetes.io/projected/716a3520-bede-48e6-9a9c-34c053b33ec0-kube-api-access-k5grq\") pod \"maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:11.952389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.952368 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6shm\" (UniqueName: \"kubernetes.io/projected/2bec1809-a925-451b-9d14-7bdf5377bfb3-kube-api-access-b6shm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:11.952488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.952394 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:11.952488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:11.952409 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bec1809-a925-451b-9d14-7bdf5377bfb3-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:12.054087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:12.054038 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:12.054316 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:12.054198 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5grq\" (UniqueName: \"kubernetes.io/projected/716a3520-bede-48e6-9a9c-34c053b33ec0-kube-api-access-k5grq\") pod \"maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:12.054399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:12.054376 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:12.055147 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:12.055120 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:12.056898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:12.056872 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:12.078118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:12.078083 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5grq\" (UniqueName: \"kubernetes.io/projected/716a3520-bede-48e6-9a9c-34c053b33ec0-kube-api-access-k5grq\") pod \"maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:12.222878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:12.222796 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:12.448994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:12.448966 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" Oct 30 06:53:12.448994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:12.448984 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn" event={"ID":"2bec1809-a925-451b-9d14-7bdf5377bfb3","Type":"ContainerDied","Data":"4625e8cb222f05485f2485fd951d734df39ccb9b97afb2d0d2150f21a9a534ca"} Oct 30 06:53:12.449220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:12.449027 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4625e8cb222f05485f2485fd951d734df39ccb9b97afb2d0d2150f21a9a534ca" Oct 30 06:53:12.539121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:12.539086 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5"] Oct 30 06:53:12.540503 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:53:12.540474 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716a3520_bede_48e6_9a9c_34c053b33ec0.slice/crio-fcafa8cf52c820c5dee997f6f020dd863a4fa565e4a8f36f09cf3cfcbd645f29 WatchSource:0}: Error finding container fcafa8cf52c820c5dee997f6f020dd863a4fa565e4a8f36f09cf3cfcbd645f29: Status 404 returned error can't find the container with id fcafa8cf52c820c5dee997f6f020dd863a4fa565e4a8f36f09cf3cfcbd645f29 Oct 30 06:53:13.460494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:13.460454 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" event={"ID":"716a3520-bede-48e6-9a9c-34c053b33ec0","Type":"ContainerStarted","Data":"503312f059b8a987298c3f1c5dfa7cac086a596b2e34c2589709b6dbfb61512c"} Oct 30 06:53:13.460918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:13.460515 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" event={"ID":"716a3520-bede-48e6-9a9c-34c053b33ec0","Type":"ContainerStarted","Data":"fcafa8cf52c820c5dee997f6f020dd863a4fa565e4a8f36f09cf3cfcbd645f29"} Oct 30 06:53:13.503631 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:13.503577 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" podStartSLOduration=2.503562837 podStartE2EDuration="2.503562837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:53:13.499005786 +0000 UTC m=+3607.494962796" watchObservedRunningTime="2026-10-30 06:53:13.503562837 +0000 UTC m=+3607.499519865" Oct 30 06:53:14.254344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:14.254293 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:14.267028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:14.266993 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:14.474857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:14.474825 5001 generic.go:356] "Generic (PLEG): container finished" podID="716a3520-bede-48e6-9a9c-34c053b33ec0" containerID="503312f059b8a987298c3f1c5dfa7cac086a596b2e34c2589709b6dbfb61512c" exitCode=0 Oct 30 06:53:14.475370 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:14.474915 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" event={"ID":"716a3520-bede-48e6-9a9c-34c053b33ec0","Type":"ContainerDied","Data":"503312f059b8a987298c3f1c5dfa7cac086a596b2e34c2589709b6dbfb61512c"} Oct 30 06:53:14.486802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:14.486777 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:53:15.706525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:15.706491 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:15.841613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:15.841523 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-openshift-ca\") pod \"716a3520-bede-48e6-9a9c-34c053b33ec0\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " Oct 30 06:53:15.841613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:15.841580 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716a3520-bede-48e6-9a9c-34c053b33ec0-kube-api-access-k5grq\" (UniqueName: \"kubernetes.io/projected/716a3520-bede-48e6-9a9c-34c053b33ec0-kube-api-access-k5grq\") pod \"716a3520-bede-48e6-9a9c-34c053b33ec0\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " Oct 30 06:53:15.841836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:15.841646 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-xks-ca\") pod \"716a3520-bede-48e6-9a9c-34c053b33ec0\" (UID: \"716a3520-bede-48e6-9a9c-34c053b33ec0\") " Oct 30 06:53:15.841908 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:15.841875 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-openshift-ca" pod "716a3520-bede-48e6-9a9c-34c053b33ec0" (UID: "716a3520-bede-48e6-9a9c-34c053b33ec0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:15.842529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:15.842407 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:15.843987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:15.843961 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-xks-ca" pod "716a3520-bede-48e6-9a9c-34c053b33ec0" (UID: "716a3520-bede-48e6-9a9c-34c053b33ec0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:15.844083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:15.844038 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716a3520-bede-48e6-9a9c-34c053b33ec0-kube-api-access-k5grq" pod "716a3520-bede-48e6-9a9c-34c053b33ec0" (UID: "716a3520-bede-48e6-9a9c-34c053b33ec0"). InnerVolumeSpecName "kube-api-access-k5grq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:15.944574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:15.944540 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5grq\" (UniqueName: \"kubernetes.io/projected/716a3520-bede-48e6-9a9c-34c053b33ec0-kube-api-access-k5grq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:15.944574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:15.944569 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a3520-bede-48e6-9a9c-34c053b33ec0-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:16.505590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.505562 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" Oct 30 06:53:16.505752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.505562 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5" event={"ID":"716a3520-bede-48e6-9a9c-34c053b33ec0","Type":"ContainerDied","Data":"fcafa8cf52c820c5dee997f6f020dd863a4fa565e4a8f36f09cf3cfcbd645f29"} Oct 30 06:53:16.505752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.505698 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcafa8cf52c820c5dee997f6f020dd863a4fa565e4a8f36f09cf3cfcbd645f29" Oct 30 06:53:16.507752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.507725 5001 generic.go:356] "Generic (PLEG): container finished" podID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerID="f6080c7fb47873b68291a97b3faf495e9b3302bc6ea7a31767abc7392bc5af79" exitCode=0 Oct 30 06:53:16.507875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.507785 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" event={"ID":"75ca4d25-1038-4aaa-9d74-436b3222223c","Type":"ContainerDied","Data":"f6080c7fb47873b68291a97b3faf495e9b3302bc6ea7a31767abc7392bc5af79"} Oct 30 06:53:16.507875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.507840 5001 scope.go:122] "RemoveContainer" containerID="2e78a0807bb4d0fbc21fdf342360b4118d5881bbe47f5789e89cbb7d93d6bebe" Oct 30 06:53:16.531788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.531765 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:53:16.656528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.656433 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ca4d25-1038-4aaa-9d74-436b3222223c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ca4d25-1038-4aaa-9d74-436b3222223c-istiod-ca-cert\") pod \"75ca4d25-1038-4aaa-9d74-436b3222223c\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " Oct 30 06:53:16.656785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.656767 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-data\") pod \"75ca4d25-1038-4aaa-9d74-436b3222223c\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " Oct 30 06:53:16.657059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.657045 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-envoy\") pod \"75ca4d25-1038-4aaa-9d74-436b3222223c\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " Oct 30 06:53:16.657755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.657729 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-token\" (UniqueName: \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-token\") pod \"75ca4d25-1038-4aaa-9d74-436b3222223c\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " Oct 30 06:53:16.658091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.658044 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-kube-api-access-wcvbw\" (UniqueName: \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-kube-api-access-wcvbw\") pod \"75ca4d25-1038-4aaa-9d74-436b3222223c\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " Oct 30 06:53:16.660282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.660260 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-credential-socket\") pod \"75ca4d25-1038-4aaa-9d74-436b3222223c\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " Oct 30 06:53:16.660478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.660460 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-certs\") pod \"75ca4d25-1038-4aaa-9d74-436b3222223c\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " Oct 30 06:53:16.661612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.661593 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-socket\") pod \"75ca4d25-1038-4aaa-9d74-436b3222223c\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " Oct 30 06:53:16.661786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.658570 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-data" pod "75ca4d25-1038-4aaa-9d74-436b3222223c" (UID: "75ca4d25-1038-4aaa-9d74-436b3222223c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:16.661847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.659488 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ca4d25-1038-4aaa-9d74-436b3222223c-istiod-ca-cert" pod "75ca4d25-1038-4aaa-9d74-436b3222223c" (UID: "75ca4d25-1038-4aaa-9d74-436b3222223c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:16.661847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.660799 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-certs" pod "75ca4d25-1038-4aaa-9d74-436b3222223c" (UID: "75ca4d25-1038-4aaa-9d74-436b3222223c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:16.661847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.661011 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-credential-socket" pod "75ca4d25-1038-4aaa-9d74-436b3222223c" (UID: "75ca4d25-1038-4aaa-9d74-436b3222223c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:16.661847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.661013 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-envoy" pod "75ca4d25-1038-4aaa-9d74-436b3222223c" (UID: "75ca4d25-1038-4aaa-9d74-436b3222223c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:16.662126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.662101 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-socket" pod "75ca4d25-1038-4aaa-9d74-436b3222223c" (UID: "75ca4d25-1038-4aaa-9d74-436b3222223c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:16.662226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.662137 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-podinfo\") pod \"75ca4d25-1038-4aaa-9d74-436b3222223c\" (UID: \"75ca4d25-1038-4aaa-9d74-436b3222223c\") " Oct 30 06:53:16.663283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.663256 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ca4d25-1038-4aaa-9d74-436b3222223c-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:16.663465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.663285 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:16.663465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.663298 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:16.663465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.663360 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:16.663465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.663373 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:16.663465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.663388 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca4d25-1038-4aaa-9d74-436b3222223c-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:16.664126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.664097 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-kube-api-access-wcvbw" pod "75ca4d25-1038-4aaa-9d74-436b3222223c" (UID: "75ca4d25-1038-4aaa-9d74-436b3222223c"). InnerVolumeSpecName "kube-api-access-wcvbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:16.665099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.665076 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-podinfo" pod "75ca4d25-1038-4aaa-9d74-436b3222223c" (UID: "75ca4d25-1038-4aaa-9d74-436b3222223c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:53:16.666255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.666235 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-token" pod "75ca4d25-1038-4aaa-9d74-436b3222223c" (UID: "75ca4d25-1038-4aaa-9d74-436b3222223c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:16.765224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.765183 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:16.765224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.765211 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvbw\" (UniqueName: \"kubernetes.io/projected/75ca4d25-1038-4aaa-9d74-436b3222223c-kube-api-access-wcvbw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:16.765224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:16.765222 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ca4d25-1038-4aaa-9d74-436b3222223c-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:17.518925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:17.518899 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" Oct 30 06:53:17.519135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:17.518893 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6" event={"ID":"75ca4d25-1038-4aaa-9d74-436b3222223c","Type":"ContainerDied","Data":"7d4a9e26f0329c4acbda9aa2ee66f551797dcbfd469875ca49dcf0ac318e7657"} Oct 30 06:53:17.519135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:17.519034 5001 scope.go:122] "RemoveContainer" containerID="f6080c7fb47873b68291a97b3faf495e9b3302bc6ea7a31767abc7392bc5af79" Oct 30 06:53:17.616220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:17.616188 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6"] Oct 30 06:53:17.621838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:17.621807 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d2923-openshift-default-5b69647fb5-rk6m6"] Oct 30 06:53:18.120506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:18.120470 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" path="/var/lib/kubelet/pods/75ca4d25-1038-4aaa-9d74-436b3222223c/volumes" Oct 30 06:53:20.906491 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.906455 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q"] Oct 30 06:53:20.907585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.907559 5001 cpu_manager.go:414] "containerMap: removing container" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" Oct 30 06:53:20.907585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.907584 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" Oct 30 06:53:20.907766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.907596 5001 cpu_manager.go:414] "containerMap: removing container" podUID="716a3520-bede-48e6-9a9c-34c053b33ec0" containerName="revoke-keys" Oct 30 06:53:20.907766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.907603 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716a3520-bede-48e6-9a9c-34c053b33ec0" containerName="revoke-keys" Oct 30 06:53:20.907917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.907872 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" Oct 30 06:53:20.907917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.907898 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="716a3520-bede-48e6-9a9c-34c053b33ec0" containerName="revoke-keys" Oct 30 06:53:20.907917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.907914 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" Oct 30 06:53:20.913658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.913635 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:20.929533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.929504 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q"] Oct 30 06:53:20.934953 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.934931 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:53:20.935080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.935055 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:53:20.971077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.971046 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:53:20.971077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.971046 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:53:20.982888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.982847 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctsl\" (UniqueName: \"kubernetes.io/projected/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-kube-api-access-sctsl\") pod \"maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:20.983236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.983216 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:20.983398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:20.983382 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:21.084924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:21.084866 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:21.085145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:21.085038 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:21.085145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:21.085112 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctsl\" (UniqueName: \"kubernetes.io/projected/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-kube-api-access-sctsl\") pod \"maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:21.085713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:21.085677 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:21.087665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:21.087634 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:21.108448 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:21.108414 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctsl\" (UniqueName: \"kubernetes.io/projected/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-kube-api-access-sctsl\") pod \"maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:21.229982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:21.229894 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:21.556063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:21.556029 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q"] Oct 30 06:53:22.576758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:22.576717 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" event={"ID":"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d","Type":"ContainerStarted","Data":"41f21610c93e74b22b9f8b6e5e4ab63d9608390310aacbb29b0ca38e81f976d3"} Oct 30 06:53:22.577211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:22.576788 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" event={"ID":"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d","Type":"ContainerStarted","Data":"370bab565948cf6c3c97290e79c3f0cd8736abeafc80218370303cd81f4841ae"} Oct 30 06:53:22.618384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:22.618328 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" podStartSLOduration=2.618297205 podStartE2EDuration="2.618297205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:53:22.616155989 +0000 UTC m=+3616.612112999" watchObservedRunningTime="2026-10-30 06:53:22.618297205 +0000 UTC m=+3616.614254215" Oct 30 06:53:23.589139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:23.589102 5001 generic.go:356] "Generic (PLEG): container finished" podID="ab57a88d-7aa4-4a29-8130-6b2d7dc8683d" containerID="41f21610c93e74b22b9f8b6e5e4ab63d9608390310aacbb29b0ca38e81f976d3" exitCode=0 Oct 30 06:53:23.589590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:23.589171 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" event={"ID":"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d","Type":"ContainerDied","Data":"41f21610c93e74b22b9f8b6e5e4ab63d9608390310aacbb29b0ca38e81f976d3"} Oct 30 06:53:24.830499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:24.830471 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:24.881929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:24.881897 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-kube-api-access-sctsl\" (UniqueName: \"kubernetes.io/projected/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-kube-api-access-sctsl\") pod \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " Oct 30 06:53:24.882140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:24.881968 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-xks-ca\") pod \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " Oct 30 06:53:24.882140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:24.882040 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-openshift-ca\") pod \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\" (UID: \"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d\") " Oct 30 06:53:24.882487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:24.882457 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-openshift-ca" pod "ab57a88d-7aa4-4a29-8130-6b2d7dc8683d" (UID: "ab57a88d-7aa4-4a29-8130-6b2d7dc8683d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:24.883042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:24.883007 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:24.884731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:24.884706 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-kube-api-access-sctsl" pod "ab57a88d-7aa4-4a29-8130-6b2d7dc8683d" (UID: "ab57a88d-7aa4-4a29-8130-6b2d7dc8683d"). InnerVolumeSpecName "kube-api-access-sctsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:24.885060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:24.885039 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-xks-ca" pod "ab57a88d-7aa4-4a29-8130-6b2d7dc8683d" (UID: "ab57a88d-7aa4-4a29-8130-6b2d7dc8683d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:24.984943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:24.984898 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sctsl\" (UniqueName: \"kubernetes.io/projected/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-kube-api-access-sctsl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:24.984943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:24.984937 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:25.611185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:25.611155 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" Oct 30 06:53:25.611409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:25.611191 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q" event={"ID":"ab57a88d-7aa4-4a29-8130-6b2d7dc8683d","Type":"ContainerDied","Data":"370bab565948cf6c3c97290e79c3f0cd8736abeafc80218370303cd81f4841ae"} Oct 30 06:53:25.611409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:25.611241 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="370bab565948cf6c3c97290e79c3f0cd8736abeafc80218370303cd81f4841ae" Oct 30 06:53:31.703338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:31.702900 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5"] Oct 30 06:53:31.738332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:31.730922 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a0083e-zvlp5"] Oct 30 06:53:32.305500 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:32.305439 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:53:32.434506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:32.434455 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:53:32.489080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:32.487740 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" podUID="f54e188f-35ed-4d3a-b66a-9ed12b084c36" containerName="main" probeResult="failure" output="Get \"https://10.132.1.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:53:32.490673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:32.490649 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716a3520-bede-48e6-9a9c-34c053b33ec0" path="/var/lib/kubelet/pods/716a3520-bede-48e6-9a9c-34c053b33ec0/volumes" Oct 30 06:53:32.615530 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:32.615474 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 06:53:32.658337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:32.658268 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 06:53:32.755296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:32.755249 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:53:38.273549 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:53:38.273504 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:53:39.008325 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.008266 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq"] Oct 30 06:53:39.008886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.008833 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" podUID="05bc9f11-c202-425e-9382-fa1a6580b6bb" containerName="istio-proxy" containerID="cri-o://03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4" gracePeriod=30 Oct 30 06:53:39.787736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.787697 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr"] Oct 30 06:53:39.788864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.788841 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ab57a88d-7aa4-4a29-8130-6b2d7dc8683d" containerName="revoke-keys" Oct 30 06:53:39.788864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.788864 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab57a88d-7aa4-4a29-8130-6b2d7dc8683d" containerName="revoke-keys" Oct 30 06:53:39.789061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.788897 5001 cpu_manager.go:414] "containerMap: removing container" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" Oct 30 06:53:39.789061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.788904 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ca4d25-1038-4aaa-9d74-436b3222223c" containerName="istio-proxy" Oct 30 06:53:39.789184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.789075 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab57a88d-7aa4-4a29-8130-6b2d7dc8683d" containerName="revoke-keys" Oct 30 06:53:39.792941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.792913 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:39.812521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.812487 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr"] Oct 30 06:53:39.816899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.816862 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:53:39.817058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.816945 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:53:39.827107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.826522 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:39.827107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.826650 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrk8\" (UniqueName: \"kubernetes.io/projected/0ebff91a-90df-4838-8b6d-3a0902b69337-kube-api-access-rtrk8\") pod \"maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:39.827107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.826814 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:39.836114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.836080 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:53:39.836323 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.836227 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:53:39.928573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.928530 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:39.928772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.928666 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrk8\" (UniqueName: \"kubernetes.io/projected/0ebff91a-90df-4838-8b6d-3a0902b69337-kube-api-access-rtrk8\") pod \"maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:39.928830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.928818 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:39.929376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.929352 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:39.931670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.931636 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:39.951801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:39.951764 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrk8\" (UniqueName: \"kubernetes.io/projected/0ebff91a-90df-4838-8b6d-3a0902b69337-kube-api-access-rtrk8\") pod \"maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:40.112839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:40.112807 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:40.530090 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:53:40.530057 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebff91a_90df_4838_8b6d_3a0902b69337.slice/crio-0b4786b715e4a8b0ea2bce15245ada5f523faf9591e409677fd3e28d630b243c WatchSource:0}: Error finding container 0b4786b715e4a8b0ea2bce15245ada5f523faf9591e409677fd3e28d630b243c: Status 404 returned error can't find the container with id 0b4786b715e4a8b0ea2bce15245ada5f523faf9591e409677fd3e28d630b243c Oct 30 06:53:40.532218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:40.532192 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr"] Oct 30 06:53:40.616698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:40.616658 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" event={"ID":"0ebff91a-90df-4838-8b6d-3a0902b69337","Type":"ContainerStarted","Data":"0b4786b715e4a8b0ea2bce15245ada5f523faf9591e409677fd3e28d630b243c"} Oct 30 06:53:41.628940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:41.628891 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" event={"ID":"0ebff91a-90df-4838-8b6d-3a0902b69337","Type":"ContainerStarted","Data":"0a0d70da3f71307de4f911204d39d6cd04d44bd2bff9d6ce2c961f25afb34078"} Oct 30 06:53:41.675503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:41.675443 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" podStartSLOduration=2.675425251 podStartE2EDuration="2.675425251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:53:41.67029275 +0000 UTC m=+3635.666249785" watchObservedRunningTime="2026-10-30 06:53:41.675425251 +0000 UTC m=+3635.671382262" Oct 30 06:53:42.641466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:42.641434 5001 generic.go:356] "Generic (PLEG): container finished" podID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerID="58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160" exitCode=0 Oct 30 06:53:42.642045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:42.641515 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" event={"ID":"cd760fe2-e8c7-42f9-8a3f-40ec810159b4","Type":"ContainerDied","Data":"58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160"} Oct 30 06:53:42.642045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:42.641594 5001 scope.go:122] "RemoveContainer" containerID="5a2371cec333f16b59f9054c7132a50bd8827237c999ae5f964b7d337f57a303" Oct 30 06:53:42.642210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:42.642192 5001 scope.go:122] "RemoveContainer" containerID="58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160" Oct 30 06:53:42.644645 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:53:42.642676 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p_openshift-ingress(cd760fe2-e8c7-42f9-8a3f-40ec810159b4)\"" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" Oct 30 06:53:42.644645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:42.643659 5001 generic.go:356] "Generic (PLEG): container finished" podID="0ebff91a-90df-4838-8b6d-3a0902b69337" containerID="0a0d70da3f71307de4f911204d39d6cd04d44bd2bff9d6ce2c961f25afb34078" exitCode=0 Oct 30 06:53:42.644645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:42.643743 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" event={"ID":"0ebff91a-90df-4838-8b6d-3a0902b69337","Type":"ContainerDied","Data":"0a0d70da3f71307de4f911204d39d6cd04d44bd2bff9d6ce2c961f25afb34078"} Oct 30 06:53:43.869427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:43.869388 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:43.915577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:43.915488 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebff91a-90df-4838-8b6d-3a0902b69337-kube-api-access-rtrk8\" (UniqueName: \"kubernetes.io/projected/0ebff91a-90df-4838-8b6d-3a0902b69337-kube-api-access-rtrk8\") pod \"0ebff91a-90df-4838-8b6d-3a0902b69337\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " Oct 30 06:53:43.915577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:43.915528 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-openshift-ca\") pod \"0ebff91a-90df-4838-8b6d-3a0902b69337\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " Oct 30 06:53:43.915855 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:43.915690 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-xks-ca\") pod \"0ebff91a-90df-4838-8b6d-3a0902b69337\" (UID: \"0ebff91a-90df-4838-8b6d-3a0902b69337\") " Oct 30 06:53:43.916003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:43.915980 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-openshift-ca" pod "0ebff91a-90df-4838-8b6d-3a0902b69337" (UID: "0ebff91a-90df-4838-8b6d-3a0902b69337"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:43.916458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:43.916436 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:43.918031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:43.917999 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-xks-ca" pod "0ebff91a-90df-4838-8b6d-3a0902b69337" (UID: "0ebff91a-90df-4838-8b6d-3a0902b69337"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:43.918284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:43.918260 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebff91a-90df-4838-8b6d-3a0902b69337-kube-api-access-rtrk8" pod "0ebff91a-90df-4838-8b6d-3a0902b69337" (UID: "0ebff91a-90df-4838-8b6d-3a0902b69337"). InnerVolumeSpecName "kube-api-access-rtrk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:44.018967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.018926 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtrk8\" (UniqueName: \"kubernetes.io/projected/0ebff91a-90df-4838-8b6d-3a0902b69337-kube-api-access-rtrk8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.018967 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.018968 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebff91a-90df-4838-8b6d-3a0902b69337-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.334273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.334243 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:53:44.423615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.423584 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q"] Oct 30 06:53:44.424710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.424280 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-credential-socket\") pod \"05bc9f11-c202-425e-9382-fa1a6580b6bb\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " Oct 30 06:53:44.424898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.424721 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05bc9f11-c202-425e-9382-fa1a6580b6bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bc9f11-c202-425e-9382-fa1a6580b6bb-istiod-ca-cert\") pod \"05bc9f11-c202-425e-9382-fa1a6580b6bb\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " Oct 30 06:53:44.424898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.424593 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-credential-socket" pod "05bc9f11-c202-425e-9382-fa1a6580b6bb" (UID: "05bc9f11-c202-425e-9382-fa1a6580b6bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:44.424898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.424785 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-socket\") pod \"05bc9f11-c202-425e-9382-fa1a6580b6bb\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " Oct 30 06:53:44.424898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.424854 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-certs\") pod \"05bc9f11-c202-425e-9382-fa1a6580b6bb\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " Oct 30 06:53:44.425188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.424919 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-data\") pod \"05bc9f11-c202-425e-9382-fa1a6580b6bb\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " Oct 30 06:53:44.425188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.424975 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-podinfo\") pod \"05bc9f11-c202-425e-9382-fa1a6580b6bb\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " Oct 30 06:53:44.425188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.425142 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-kube-api-access-c2ksg\" (UniqueName: \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-kube-api-access-c2ksg\") pod \"05bc9f11-c202-425e-9382-fa1a6580b6bb\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " Oct 30 06:53:44.425188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.425167 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05bc9f11-c202-425e-9382-fa1a6580b6bb-istiod-ca-cert" pod "05bc9f11-c202-425e-9382-fa1a6580b6bb" (UID: "05bc9f11-c202-425e-9382-fa1a6580b6bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:44.425446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.425194 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-token\" (UniqueName: \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-token\") pod \"05bc9f11-c202-425e-9382-fa1a6580b6bb\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " Oct 30 06:53:44.425446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.425197 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-socket" pod "05bc9f11-c202-425e-9382-fa1a6580b6bb" (UID: "05bc9f11-c202-425e-9382-fa1a6580b6bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:44.425446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.425253 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-envoy\") pod \"05bc9f11-c202-425e-9382-fa1a6580b6bb\" (UID: \"05bc9f11-c202-425e-9382-fa1a6580b6bb\") " Oct 30 06:53:44.425619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.425472 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-certs" pod "05bc9f11-c202-425e-9382-fa1a6580b6bb" (UID: "05bc9f11-c202-425e-9382-fa1a6580b6bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:44.426197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.426054 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-data" pod "05bc9f11-c202-425e-9382-fa1a6580b6bb" (UID: "05bc9f11-c202-425e-9382-fa1a6580b6bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:44.426197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.426070 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bc9f11-c202-425e-9382-fa1a6580b6bb-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.426197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.426095 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.426197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.426108 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.426197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.426122 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.428103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.428078 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-podinfo" pod "05bc9f11-c202-425e-9382-fa1a6580b6bb" (UID: "05bc9f11-c202-425e-9382-fa1a6580b6bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:53:44.428327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.428282 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-token" pod "05bc9f11-c202-425e-9382-fa1a6580b6bb" (UID: "05bc9f11-c202-425e-9382-fa1a6580b6bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:44.428416 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.428380 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-envoy" pod "05bc9f11-c202-425e-9382-fa1a6580b6bb" (UID: "05bc9f11-c202-425e-9382-fa1a6580b6bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:44.428522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.428503 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-kube-api-access-c2ksg" pod "05bc9f11-c202-425e-9382-fa1a6580b6bb" (UID: "05bc9f11-c202-425e-9382-fa1a6580b6bb"). InnerVolumeSpecName "kube-api-access-c2ksg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:44.527340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.527242 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.527340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.527274 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.527340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.527284 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2ksg\" (UniqueName: \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-kube-api-access-c2ksg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.527340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.527294 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.527340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.527318 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bc9f11-c202-425e-9382-fa1a6580b6bb-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.670651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.670621 5001 generic.go:356] "Generic (PLEG): container finished" podID="05bc9f11-c202-425e-9382-fa1a6580b6bb" containerID="03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4" exitCode=0 Oct 30 06:53:44.670848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.670820 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" Oct 30 06:53:44.671974 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.671935 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" event={"ID":"05bc9f11-c202-425e-9382-fa1a6580b6bb","Type":"ContainerDied","Data":"03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4"} Oct 30 06:53:44.672149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.672004 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq" event={"ID":"05bc9f11-c202-425e-9382-fa1a6580b6bb","Type":"ContainerDied","Data":"d5c101a3c0c1f12596fe7c1c4ddf1c987d51c341d20627cb52f0949c5d62007e"} Oct 30 06:53:44.672149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.672036 5001 scope.go:122] "RemoveContainer" containerID="03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4" Oct 30 06:53:44.675140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.674343 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" event={"ID":"0ebff91a-90df-4838-8b6d-3a0902b69337","Type":"ContainerDied","Data":"0b4786b715e4a8b0ea2bce15245ada5f523faf9591e409677fd3e28d630b243c"} Oct 30 06:53:44.675140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.674406 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b4786b715e4a8b0ea2bce15245ada5f523faf9591e409677fd3e28d630b243c" Oct 30 06:53:44.675140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.674356 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr" Oct 30 06:53:44.689641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.689616 5001 scope.go:122] "RemoveContainer" containerID="03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4" Oct 30 06:53:44.690020 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:53:44.689998 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4\": container with ID starting with 03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4 not found: ID does not exist" containerID="03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4" Oct 30 06:53:44.690106 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.690031 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4"} err="failed to get container status \"03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4\": rpc error: code = NotFound desc = could not find container \"03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4\": container with ID starting with 03a13162da9081934ee25550fc811c77d12c9f5dcf347c2c43955e080c3385f4 not found: ID does not exist" Oct 30 06:53:44.737133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.737096 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq"] Oct 30 06:53:44.742002 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:44.741978 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-rcpwq"] Oct 30 06:53:46.119775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:46.119731 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05bc9f11-c202-425e-9382-fa1a6580b6bb" path="/var/lib/kubelet/pods/05bc9f11-c202-425e-9382-fa1a6580b6bb/volumes" Oct 30 06:53:49.005457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:49.005376 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:53:49.005956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:49.005938 5001 scope.go:122] "RemoveContainer" containerID="58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160" Oct 30 06:53:49.006296 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:53:49.006274 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p_openshift-ingress(cd760fe2-e8c7-42f9-8a3f-40ec810159b4)\"" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" Oct 30 06:53:50.239738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:50.239700 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd"] Oct 30 06:53:50.272157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:50.272120 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9"] Oct 30 06:53:54.110632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:54.110598 5001 scope.go:122] "RemoveContainer" containerID="58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160" Oct 30 06:53:54.113068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:54.113032 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:53:54.113156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:54.113126 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:53:54.113203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:54.113172 5001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:53:54.795272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:54.795238 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" event={"ID":"cd760fe2-e8c7-42f9-8a3f-40ec810159b4","Type":"ContainerStarted","Data":"57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9"} Oct 30 06:53:54.981544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:54.981495 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:53:55.241021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.240953 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" podUID="2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb" containerName="payload-processing" containerID="cri-o://e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94" gracePeriod=25 Oct 30 06:53:55.274987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.273410 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" podUID="a315417d-889e-448b-b3fd-ebf54e93aac1" containerName="payload-pre-processing" containerID="cri-o://36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189" gracePeriod=25 Oct 30 06:53:55.644876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.644828 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:53:55.726933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.726911 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:53:55.742273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.741264 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a315417d-889e-448b-b3fd-ebf54e93aac1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a315417d-889e-448b-b3fd-ebf54e93aac1-plugins-config-volume\") pod \"a315417d-889e-448b-b3fd-ebf54e93aac1\" (UID: \"a315417d-889e-448b-b3fd-ebf54e93aac1\") " Oct 30 06:53:55.742273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.741399 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a315417d-889e-448b-b3fd-ebf54e93aac1-kube-api-access-kwqv7\" (UniqueName: \"kubernetes.io/projected/a315417d-889e-448b-b3fd-ebf54e93aac1-kube-api-access-kwqv7\") pod \"a315417d-889e-448b-b3fd-ebf54e93aac1\" (UID: \"a315417d-889e-448b-b3fd-ebf54e93aac1\") " Oct 30 06:53:55.742698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.742667 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a315417d-889e-448b-b3fd-ebf54e93aac1-plugins-config-volume" pod "a315417d-889e-448b-b3fd-ebf54e93aac1" (UID: "a315417d-889e-448b-b3fd-ebf54e93aac1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:55.744827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.744791 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a315417d-889e-448b-b3fd-ebf54e93aac1-kube-api-access-kwqv7" pod "a315417d-889e-448b-b3fd-ebf54e93aac1" (UID: "a315417d-889e-448b-b3fd-ebf54e93aac1"). InnerVolumeSpecName "kube-api-access-kwqv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:55.811255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.811160 5001 generic.go:356] "Generic (PLEG): container finished" podID="2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb" containerID="e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94" exitCode=0 Oct 30 06:53:55.811465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.811352 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" event={"ID":"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb","Type":"ContainerDied","Data":"e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94"} Oct 30 06:53:55.811465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.811417 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" event={"ID":"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb","Type":"ContainerDied","Data":"62f0c72b6c77426910e6c3323aab7e7361e6872b18570419366052435e021e74"} Oct 30 06:53:55.811465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.811447 5001 scope.go:122] "RemoveContainer" containerID="e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94" Oct 30 06:53:55.811671 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.811631 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd" Oct 30 06:53:55.814494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.814472 5001 generic.go:356] "Generic (PLEG): container finished" podID="a315417d-889e-448b-b3fd-ebf54e93aac1" containerID="36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189" exitCode=0 Oct 30 06:53:55.815613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.815592 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" Oct 30 06:53:55.820427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.820396 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" event={"ID":"a315417d-889e-448b-b3fd-ebf54e93aac1","Type":"ContainerDied","Data":"36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189"} Oct 30 06:53:55.820553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.820454 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9" event={"ID":"a315417d-889e-448b-b3fd-ebf54e93aac1","Type":"ContainerDied","Data":"7336f0d4f31ccea545e69745ae6eaee10f7213943ec27293f54344c5c130dc86"} Oct 30 06:53:55.832726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.832703 5001 scope.go:122] "RemoveContainer" containerID="e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94" Oct 30 06:53:55.833349 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:53:55.833270 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94\": container with ID starting with e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94 not found: ID does not exist" containerID="e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94" Oct 30 06:53:55.833349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.833330 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94"} err="failed to get container status \"e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94\": rpc error: code = NotFound desc = could not find container \"e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94\": container with ID starting with e4beeb7fbfb8f180262c9764d10112eae5acb822d818b3af437601b754e7bd94 not found: ID does not exist" Oct 30 06:53:55.833588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.833355 5001 scope.go:122] "RemoveContainer" containerID="36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189" Oct 30 06:53:55.843251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.843222 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-plugins-config-volume\") pod \"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb\" (UID: \"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb\") " Oct 30 06:53:55.843514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.843490 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-kube-api-access-wr2jw\" (UniqueName: \"kubernetes.io/projected/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-kube-api-access-wr2jw\") pod \"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb\" (UID: \"2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb\") " Oct 30 06:53:55.844747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.844714 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a315417d-889e-448b-b3fd-ebf54e93aac1-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:55.844846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.844762 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwqv7\" (UniqueName: \"kubernetes.io/projected/a315417d-889e-448b-b3fd-ebf54e93aac1-kube-api-access-kwqv7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:55.846948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.846903 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-plugins-config-volume" pod "2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb" (UID: "2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:55.849835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.849809 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-kube-api-access-wr2jw" pod "2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb" (UID: "2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb"). InnerVolumeSpecName "kube-api-access-wr2jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:55.850236 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.850214 5001 scope.go:122] "RemoveContainer" containerID="36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189" Oct 30 06:53:55.850671 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:53:55.850646 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189\": container with ID starting with 36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189 not found: ID does not exist" containerID="36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189" Oct 30 06:53:55.850769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.850680 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189"} err="failed to get container status \"36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189\": rpc error: code = NotFound desc = could not find container \"36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189\": container with ID starting with 36df292d58465aefd07d738787a523e7cf3e5e6ae25889f84d4c144e270f0189 not found: ID does not exist" Oct 30 06:53:55.906255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.906209 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9"] Oct 30 06:53:55.911285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.911254 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-eaee88-688ff65ddc-zq8v9"] Oct 30 06:53:55.946730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.946703 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr2jw\" (UniqueName: \"kubernetes.io/projected/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-kube-api-access-wr2jw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:55.946730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.946731 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:53:55.982473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.982436 5001 patch_prober.go:29] interesting pod/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 06:53:55.982665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:55.982506 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 06:53:56.123592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:56.123560 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a315417d-889e-448b-b3fd-ebf54e93aac1" path="/var/lib/kubelet/pods/a315417d-889e-448b-b3fd-ebf54e93aac1/volumes" Oct 30 06:53:56.187729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:56.187604 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd"] Oct 30 06:53:56.193134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:56.193100 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-eaee88-544ddb87d6-cckhd"] Oct 30 06:53:56.635624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:56.635592 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc"] Oct 30 06:53:56.636107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:56.635794 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" containerName="istio-proxy" containerID="cri-o://877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94" gracePeriod=30 Oct 30 06:53:56.981922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:56.981830 5001 patch_prober.go:29] interesting pod/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 06:53:56.981922 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:56.981897 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 06:53:57.779368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:57.779326 5001 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:53:57.779871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:57.779408 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:53:57.981828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:57.981780 5001 patch_prober.go:29] interesting pod/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 06:53:57.982055 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:57.981851 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 06:53:58.119704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.119669 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb" path="/var/lib/kubelet/pods/2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb/volumes" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.946027 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl"] Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947462 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb" containerName="payload-processing" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947483 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb" containerName="payload-processing" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947537 5001 cpu_manager.go:414] "containerMap: removing container" podUID="05bc9f11-c202-425e-9382-fa1a6580b6bb" containerName="istio-proxy" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947546 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05bc9f11-c202-425e-9382-fa1a6580b6bb" containerName="istio-proxy" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947597 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a315417d-889e-448b-b3fd-ebf54e93aac1" containerName="payload-pre-processing" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947605 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a315417d-889e-448b-b3fd-ebf54e93aac1" containerName="payload-pre-processing" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947638 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0ebff91a-90df-4838-8b6d-3a0902b69337" containerName="revoke-keys" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947645 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebff91a-90df-4838-8b6d-3a0902b69337" containerName="revoke-keys" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947861 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a315417d-889e-448b-b3fd-ebf54e93aac1" containerName="payload-pre-processing" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947886 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebff91a-90df-4838-8b6d-3a0902b69337" containerName="revoke-keys" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947901 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1311fe-c94d-4cbf-a6fe-5e9bfb15cbfb" containerName="payload-processing" Oct 30 06:53:58.948340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.947920 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="05bc9f11-c202-425e-9382-fa1a6580b6bb" containerName="istio-proxy" Oct 30 06:53:58.959154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.959123 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:53:58.978595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.977977 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:53:58.978595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.978104 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:53:58.978595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.978182 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:53:58.998075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.992457 5001 patch_prober.go:29] interesting pod/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 06:53:58.998075 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.992554 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 06:53:59.002373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:58.999015 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:53:59.035032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.034340 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5ws\" (UniqueName: \"kubernetes.io/projected/39709b3b-ac31-48ce-8399-6c59101d4a32-kube-api-access-ks5ws\") pod \"maas-api-revoke-sub-9c3a8846-9d8dl\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:53:59.035032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.034521 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3a8846-9d8dl\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:53:59.035032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.034784 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3a8846-9d8dl\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:53:59.057401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.057363 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc"] Oct 30 06:53:59.097644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.097611 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:53:59.120733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.120701 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl"] Oct 30 06:53:59.140839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.140798 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1740849-kkjnc\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:53:59.141399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.141367 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5ws\" (UniqueName: \"kubernetes.io/projected/39709b3b-ac31-48ce-8399-6c59101d4a32-kube-api-access-ks5ws\") pod \"maas-api-revoke-sub-9c3a8846-9d8dl\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:53:59.143554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.141583 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rqr7\" (UniqueName: \"kubernetes.io/projected/53abaaf4-3338-48a9-8834-a24490e47beb-kube-api-access-9rqr7\") pod \"maas-api-revoke-sub-e1740849-kkjnc\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:53:59.143554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.141709 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3a8846-9d8dl\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:53:59.143554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.141924 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3a8846-9d8dl\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:53:59.143554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.141955 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1740849-kkjnc\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:53:59.143554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.143208 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3a8846-9d8dl\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:53:59.170896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.167161 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3a8846-9d8dl\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:53:59.177337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.175686 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5ws\" (UniqueName: \"kubernetes.io/projected/39709b3b-ac31-48ce-8399-6c59101d4a32-kube-api-access-ks5ws\") pod \"maas-api-revoke-sub-9c3a8846-9d8dl\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:53:59.212584 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.212479 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc"] Oct 30 06:53:59.322641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.322601 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:53:59.324365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.324329 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1740849-kkjnc\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:53:59.324501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.324477 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1740849-kkjnc\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:53:59.324564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.324532 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rqr7\" (UniqueName: \"kubernetes.io/projected/53abaaf4-3338-48a9-8834-a24490e47beb-kube-api-access-9rqr7\") pod \"maas-api-revoke-sub-e1740849-kkjnc\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:53:59.327022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.326299 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1740849-kkjnc\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:53:59.339681 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.339653 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1740849-kkjnc\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:53:59.348496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.348428 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rqr7\" (UniqueName: \"kubernetes.io/projected/53abaaf4-3338-48a9-8834-a24490e47beb-kube-api-access-9rqr7\") pod \"maas-api-revoke-sub-e1740849-kkjnc\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:53:59.403771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.403645 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q"] Oct 30 06:53:59.412030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.411998 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c148a-lcf8q"] Oct 30 06:53:59.464392 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.437615 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:53:59.512352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.489411 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" containerID="cri-o://cc2bb5da0edfd84c93be543edacd0e19d33838512803360af7b59fb0cce2045a" gracePeriod=45 Oct 30 06:53:59.643125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.643086 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": dial tcp 10.132.0.85:8000: connect: connection refused" Oct 30 06:53:59.862161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.851135 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl"] Oct 30 06:53:59.904997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.904968 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc"] Oct 30 06:53:59.910714 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:53:59.910683 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53abaaf4_3338_48a9_8834_a24490e47beb.slice/crio-aa380e815788c9ccfb6a25134211c67b1a494ac63dfb39385a2243dccf79a7b8 WatchSource:0}: Error finding container aa380e815788c9ccfb6a25134211c67b1a494ac63dfb39385a2243dccf79a7b8: Status 404 returned error can't find the container with id aa380e815788c9ccfb6a25134211c67b1a494ac63dfb39385a2243dccf79a7b8 Oct 30 06:53:59.988442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.988405 5001 patch_prober.go:29] interesting pod/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" start-of-body= Oct 30 06:53:59.988913 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.988466 5001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.162:15021/healthz/ready\": dial tcp 10.132.1.162:15021: connect: connection refused" Oct 30 06:53:59.990504 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.990475 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" event={"ID":"53abaaf4-3338-48a9-8834-a24490e47beb","Type":"ContainerStarted","Data":"aa380e815788c9ccfb6a25134211c67b1a494ac63dfb39385a2243dccf79a7b8"} Oct 30 06:53:59.994273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.994251 5001 generic.go:356] "Generic (PLEG): container finished" podID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerID="cc2bb5da0edfd84c93be543edacd0e19d33838512803360af7b59fb0cce2045a" exitCode=0 Oct 30 06:53:59.994668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.994647 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" event={"ID":"8e8112df-d5d4-41e4-adb2-560640e069b9","Type":"ContainerDied","Data":"cc2bb5da0edfd84c93be543edacd0e19d33838512803360af7b59fb0cce2045a"} Oct 30 06:53:59.996480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:53:59.996454 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" event={"ID":"39709b3b-ac31-48ce-8399-6c59101d4a32","Type":"ContainerStarted","Data":"4dfb1e63758b658d7fbaccb218fe9b7b6ad01eb7006f7919a9802cae400c86d8"} Oct 30 06:54:00.095987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.095417 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:54:00.127253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.127214 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab57a88d-7aa4-4a29-8130-6b2d7dc8683d" path="/var/lib/kubelet/pods/ab57a88d-7aa4-4a29-8130-6b2d7dc8683d/volumes" Oct 30 06:54:00.177747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.176527 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-home\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-home\") pod \"8e8112df-d5d4-41e4-adb2-560640e069b9\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " Oct 30 06:54:00.177747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.176578 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-tmp-dir\") pod \"8e8112df-d5d4-41e4-adb2-560640e069b9\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " Oct 30 06:54:00.177747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.176625 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-dshm\") pod \"8e8112df-d5d4-41e4-adb2-560640e069b9\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " Oct 30 06:54:00.177747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.176778 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e8112df-d5d4-41e4-adb2-560640e069b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8112df-d5d4-41e4-adb2-560640e069b9-tls-certs\") pod \"8e8112df-d5d4-41e4-adb2-560640e069b9\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " Oct 30 06:54:00.177747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.176959 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-model-cache\") pod \"8e8112df-d5d4-41e4-adb2-560640e069b9\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " Oct 30 06:54:00.177747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.176995 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8112df-d5d4-41e4-adb2-560640e069b9-kube-api-access-szzjk\" (UniqueName: \"kubernetes.io/projected/8e8112df-d5d4-41e4-adb2-560640e069b9-kube-api-access-szzjk\") pod \"8e8112df-d5d4-41e4-adb2-560640e069b9\" (UID: \"8e8112df-d5d4-41e4-adb2-560640e069b9\") " Oct 30 06:54:00.184400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.184366 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-home" pod "8e8112df-d5d4-41e4-adb2-560640e069b9" (UID: "8e8112df-d5d4-41e4-adb2-560640e069b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:00.192595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.192556 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-model-cache" pod "8e8112df-d5d4-41e4-adb2-560640e069b9" (UID: "8e8112df-d5d4-41e4-adb2-560640e069b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:00.193561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.193531 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-tmp-dir" pod "8e8112df-d5d4-41e4-adb2-560640e069b9" (UID: "8e8112df-d5d4-41e4-adb2-560640e069b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:00.201001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.200962 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-dshm" pod "8e8112df-d5d4-41e4-adb2-560640e069b9" (UID: "8e8112df-d5d4-41e4-adb2-560640e069b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:00.201775 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.201745 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8112df-d5d4-41e4-adb2-560640e069b9-tls-certs" pod "8e8112df-d5d4-41e4-adb2-560640e069b9" (UID: "8e8112df-d5d4-41e4-adb2-560640e069b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:54:00.215362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.214104 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8112df-d5d4-41e4-adb2-560640e069b9-kube-api-access-szzjk" pod "8e8112df-d5d4-41e4-adb2-560640e069b9" (UID: "8e8112df-d5d4-41e4-adb2-560640e069b9"). InnerVolumeSpecName "kube-api-access-szzjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:00.295337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.294192 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.295337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.294233 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szzjk\" (UniqueName: \"kubernetes.io/projected/8e8112df-d5d4-41e4-adb2-560640e069b9-kube-api-access-szzjk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.295337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.294248 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.295337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.294264 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.295337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.294277 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8112df-d5d4-41e4-adb2-560640e069b9-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.295337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:00.294289 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8112df-d5d4-41e4-adb2-560640e069b9-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:01.007497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.007461 5001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:54:01.008025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.007995 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:54:01.009246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.009221 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:54:01.011176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.011142 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" event={"ID":"53abaaf4-3338-48a9-8834-a24490e47beb","Type":"ContainerStarted","Data":"ef3712b296b4eb50ad980c4058413e22a5b9118a862f949fd77351c47bc94199"} Oct 30 06:54:01.014567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.013835 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" Oct 30 06:54:01.014567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.013865 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q" event={"ID":"8e8112df-d5d4-41e4-adb2-560640e069b9","Type":"ContainerDied","Data":"2723a1483f25293cf9daccbb1e7d235085bdf882a43941b95c13f1d76ffa05e9"} Oct 30 06:54:01.014567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.013924 5001 scope.go:122] "RemoveContainer" containerID="cc2bb5da0edfd84c93be543edacd0e19d33838512803360af7b59fb0cce2045a" Oct 30 06:54:01.016538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.016514 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" event={"ID":"39709b3b-ac31-48ce-8399-6c59101d4a32","Type":"ContainerStarted","Data":"8f8d8a72b497c19107c781c6352d12241667d37cb9ad670208d67ac5111c8ba8"} Oct 30 06:54:01.153376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.153268 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q"] Oct 30 06:54:01.177133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.177096 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-2-w4-eaee88-kserve-845bcb4b94-zj26q"] Oct 30 06:54:01.217896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.217847 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" podStartSLOduration=3.217831848 podStartE2EDuration="3.217831848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:54:01.215468444 +0000 UTC m=+3655.211425472" watchObservedRunningTime="2026-10-30 06:54:01.217831848 +0000 UTC m=+3655.213788858" Oct 30 06:54:01.260148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.260023 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" podStartSLOduration=3.260002948 podStartE2EDuration="3.260002948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:54:01.253976204 +0000 UTC m=+3655.249933222" watchObservedRunningTime="2026-10-30 06:54:01.260002948 +0000 UTC m=+3655.255960046" Oct 30 06:54:01.944040 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.943998 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:54:01.971435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.971399 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-envoy\") pod \"e24b04da-c878-487b-8805-18e48cfc7f37\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " Oct 30 06:54:01.971660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.971444 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-kube-api-access-db4wb\" (UniqueName: \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-kube-api-access-db4wb\") pod \"e24b04da-c878-487b-8805-18e48cfc7f37\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " Oct 30 06:54:01.971660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.971611 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e24b04da-c878-487b-8805-18e48cfc7f37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24b04da-c878-487b-8805-18e48cfc7f37-istio-podinfo\") pod \"e24b04da-c878-487b-8805-18e48cfc7f37\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " Oct 30 06:54:01.971660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.971655 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-data\") pod \"e24b04da-c878-487b-8805-18e48cfc7f37\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " Oct 30 06:54:01.971865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.971832 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-istio-token\" (UniqueName: \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-istio-token\") pod \"e24b04da-c878-487b-8805-18e48cfc7f37\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " Oct 30 06:54:01.971988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.971968 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-certs\") pod \"e24b04da-c878-487b-8805-18e48cfc7f37\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " Oct 30 06:54:01.972059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.972009 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-credential-socket\") pod \"e24b04da-c878-487b-8805-18e48cfc7f37\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " Oct 30 06:54:01.972118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.972082 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e24b04da-c878-487b-8805-18e48cfc7f37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24b04da-c878-487b-8805-18e48cfc7f37-istiod-ca-cert\") pod \"e24b04da-c878-487b-8805-18e48cfc7f37\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " Oct 30 06:54:01.972178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.972119 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-socket\") pod \"e24b04da-c878-487b-8805-18e48cfc7f37\" (UID: \"e24b04da-c878-487b-8805-18e48cfc7f37\") " Oct 30 06:54:01.972287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.972003 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-data" pod "e24b04da-c878-487b-8805-18e48cfc7f37" (UID: "e24b04da-c878-487b-8805-18e48cfc7f37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:01.972287 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.972205 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-certs" pod "e24b04da-c878-487b-8805-18e48cfc7f37" (UID: "e24b04da-c878-487b-8805-18e48cfc7f37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:01.972774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.972571 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24b04da-c878-487b-8805-18e48cfc7f37-istiod-ca-cert" pod "e24b04da-c878-487b-8805-18e48cfc7f37" (UID: "e24b04da-c878-487b-8805-18e48cfc7f37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:01.972991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.972945 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-socket" pod "e24b04da-c878-487b-8805-18e48cfc7f37" (UID: "e24b04da-c878-487b-8805-18e48cfc7f37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:01.973226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.973095 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:01.973226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.973123 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:01.973226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.973136 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24b04da-c878-487b-8805-18e48cfc7f37-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:01.973226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.973149 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:01.973606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.973579 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-credential-socket" pod "e24b04da-c878-487b-8805-18e48cfc7f37" (UID: "e24b04da-c878-487b-8805-18e48cfc7f37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:01.974208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.974183 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-envoy" pod "e24b04da-c878-487b-8805-18e48cfc7f37" (UID: "e24b04da-c878-487b-8805-18e48cfc7f37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:01.974518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.974493 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-istio-token" pod "e24b04da-c878-487b-8805-18e48cfc7f37" (UID: "e24b04da-c878-487b-8805-18e48cfc7f37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:01.974634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.974501 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-kube-api-access-db4wb" pod "e24b04da-c878-487b-8805-18e48cfc7f37" (UID: "e24b04da-c878-487b-8805-18e48cfc7f37"). InnerVolumeSpecName "kube-api-access-db4wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:01.974706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:01.974686 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e24b04da-c878-487b-8805-18e48cfc7f37-istio-podinfo" pod "e24b04da-c878-487b-8805-18e48cfc7f37" (UID: "e24b04da-c878-487b-8805-18e48cfc7f37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:54:02.044540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.044500 5001 generic.go:356] "Generic (PLEG): container finished" podID="e24b04da-c878-487b-8805-18e48cfc7f37" containerID="877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94" exitCode=0 Oct 30 06:54:02.045386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.045355 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" event={"ID":"e24b04da-c878-487b-8805-18e48cfc7f37","Type":"ContainerDied","Data":"877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94"} Oct 30 06:54:02.045640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.045622 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" event={"ID":"e24b04da-c878-487b-8805-18e48cfc7f37","Type":"ContainerDied","Data":"affccdb40031ee1d5352ba3364f115c24dc05ebdbe694e8e86c172c7911ddd92"} Oct 30 06:54:02.045777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.045763 5001 scope.go:122] "RemoveContainer" containerID="877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94" Oct 30 06:54:02.046087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.046069 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc" Oct 30 06:54:02.050665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.050634 5001 generic.go:356] "Generic (PLEG): container finished" podID="53abaaf4-3338-48a9-8834-a24490e47beb" containerID="ef3712b296b4eb50ad980c4058413e22a5b9118a862f949fd77351c47bc94199" exitCode=0 Oct 30 06:54:02.050800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.050758 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" event={"ID":"53abaaf4-3338-48a9-8834-a24490e47beb","Type":"ContainerDied","Data":"ef3712b296b4eb50ad980c4058413e22a5b9118a862f949fd77351c47bc94199"} Oct 30 06:54:02.056981 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.056949 5001 generic.go:356] "Generic (PLEG): container finished" podID="39709b3b-ac31-48ce-8399-6c59101d4a32" containerID="8f8d8a72b497c19107c781c6352d12241667d37cb9ad670208d67ac5111c8ba8" exitCode=0 Oct 30 06:54:02.057134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.057034 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" event={"ID":"39709b3b-ac31-48ce-8399-6c59101d4a32","Type":"ContainerDied","Data":"8f8d8a72b497c19107c781c6352d12241667d37cb9ad670208d67ac5111c8ba8"} Oct 30 06:54:02.071472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.071448 5001 scope.go:122] "RemoveContainer" containerID="877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94" Oct 30 06:54:02.071813 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:54:02.071787 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94\": container with ID starting with 877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94 not found: ID does not exist" containerID="877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94" Oct 30 06:54:02.072079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.071823 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94"} err="failed to get container status \"877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94\": rpc error: code = NotFound desc = could not find container \"877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94\": container with ID starting with 877dc32b534a69233ce8e184aa391d488774c5663ccf2e42630ed21c52ef8f94 not found: ID does not exist" Oct 30 06:54:02.074880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.074854 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:02.074880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.074881 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:02.075026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.074891 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24b04da-c878-487b-8805-18e48cfc7f37-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:02.075026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.074900 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db4wb\" (UniqueName: \"kubernetes.io/projected/e24b04da-c878-487b-8805-18e48cfc7f37-kube-api-access-db4wb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:02.075026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.074909 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24b04da-c878-487b-8805-18e48cfc7f37-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:02.128082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.128046 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" path="/var/lib/kubelet/pods/8e8112df-d5d4-41e4-adb2-560640e069b9/volumes" Oct 30 06:54:02.201478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.201391 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc"] Oct 30 06:54:02.209699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:02.209669 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-h4jmc"] Oct 30 06:54:03.363129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.363096 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:54:03.366476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.366411 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:54:03.410758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.410716 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-openshift-ca\") pod \"53abaaf4-3338-48a9-8834-a24490e47beb\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " Oct 30 06:54:03.410945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.410769 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-xks-ca\") pod \"53abaaf4-3338-48a9-8834-a24490e47beb\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " Oct 30 06:54:03.410945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.410882 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39709b3b-ac31-48ce-8399-6c59101d4a32-kube-api-access-ks5ws\" (UniqueName: \"kubernetes.io/projected/39709b3b-ac31-48ce-8399-6c59101d4a32-kube-api-access-ks5ws\") pod \"39709b3b-ac31-48ce-8399-6c59101d4a32\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " Oct 30 06:54:03.410945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.410914 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-xks-ca\") pod \"39709b3b-ac31-48ce-8399-6c59101d4a32\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " Oct 30 06:54:03.411134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.410962 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53abaaf4-3338-48a9-8834-a24490e47beb-kube-api-access-9rqr7\" (UniqueName: \"kubernetes.io/projected/53abaaf4-3338-48a9-8834-a24490e47beb-kube-api-access-9rqr7\") pod \"53abaaf4-3338-48a9-8834-a24490e47beb\" (UID: \"53abaaf4-3338-48a9-8834-a24490e47beb\") " Oct 30 06:54:03.411134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.411013 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-openshift-ca\") pod \"39709b3b-ac31-48ce-8399-6c59101d4a32\" (UID: \"39709b3b-ac31-48ce-8399-6c59101d4a32\") " Oct 30 06:54:03.411134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.411088 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-openshift-ca" pod "53abaaf4-3338-48a9-8834-a24490e47beb" (UID: "53abaaf4-3338-48a9-8834-a24490e47beb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:03.411951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.411913 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-openshift-ca" pod "39709b3b-ac31-48ce-8399-6c59101d4a32" (UID: "39709b3b-ac31-48ce-8399-6c59101d4a32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:03.412160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.412110 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.412160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.412144 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.413559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.413528 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-xks-ca" pod "53abaaf4-3338-48a9-8834-a24490e47beb" (UID: "53abaaf4-3338-48a9-8834-a24490e47beb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:54:03.414350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.414286 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53abaaf4-3338-48a9-8834-a24490e47beb-kube-api-access-9rqr7" pod "53abaaf4-3338-48a9-8834-a24490e47beb" (UID: "53abaaf4-3338-48a9-8834-a24490e47beb"). InnerVolumeSpecName "kube-api-access-9rqr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:03.414488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.414454 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39709b3b-ac31-48ce-8399-6c59101d4a32-kube-api-access-ks5ws" pod "39709b3b-ac31-48ce-8399-6c59101d4a32" (UID: "39709b3b-ac31-48ce-8399-6c59101d4a32"). InnerVolumeSpecName "kube-api-access-ks5ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:03.414939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.414911 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-xks-ca" pod "39709b3b-ac31-48ce-8399-6c59101d4a32" (UID: "39709b3b-ac31-48ce-8399-6c59101d4a32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:54:03.513746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.513708 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53abaaf4-3338-48a9-8834-a24490e47beb-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.513746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.513738 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks5ws\" (UniqueName: \"kubernetes.io/projected/39709b3b-ac31-48ce-8399-6c59101d4a32-kube-api-access-ks5ws\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.513746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.513748 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39709b3b-ac31-48ce-8399-6c59101d4a32-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.513746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:03.513756 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rqr7\" (UniqueName: \"kubernetes.io/projected/53abaaf4-3338-48a9-8834-a24490e47beb-kube-api-access-9rqr7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:04.121390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:04.121358 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" path="/var/lib/kubelet/pods/e24b04da-c878-487b-8805-18e48cfc7f37/volumes" Oct 30 06:54:04.124940 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:04.124903 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" Oct 30 06:54:04.125207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:04.124939 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc" event={"ID":"53abaaf4-3338-48a9-8834-a24490e47beb","Type":"ContainerDied","Data":"aa380e815788c9ccfb6a25134211c67b1a494ac63dfb39385a2243dccf79a7b8"} Oct 30 06:54:04.125299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:04.125237 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa380e815788c9ccfb6a25134211c67b1a494ac63dfb39385a2243dccf79a7b8" Oct 30 06:54:04.127004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:04.126975 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" Oct 30 06:54:04.127124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:04.127013 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl" event={"ID":"39709b3b-ac31-48ce-8399-6c59101d4a32","Type":"ContainerDied","Data":"4dfb1e63758b658d7fbaccb218fe9b7b6ad01eb7006f7919a9802cae400c86d8"} Oct 30 06:54:04.127124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:04.127068 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dfb1e63758b658d7fbaccb218fe9b7b6ad01eb7006f7919a9802cae400c86d8" Oct 30 06:54:07.265629 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.265591 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96"] Oct 30 06:54:07.266871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.266849 5001 cpu_manager.go:414] "containerMap: removing container" podUID="53abaaf4-3338-48a9-8834-a24490e47beb" containerName="revoke-keys" Oct 30 06:54:07.266871 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.266872 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53abaaf4-3338-48a9-8834-a24490e47beb" containerName="revoke-keys" Oct 30 06:54:07.267000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.266894 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" Oct 30 06:54:07.267000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.266899 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" Oct 30 06:54:07.267000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.266912 5001 cpu_manager.go:414] "containerMap: removing container" podUID="39709b3b-ac31-48ce-8399-6c59101d4a32" containerName="revoke-keys" Oct 30 06:54:07.267000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.266917 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39709b3b-ac31-48ce-8399-6c59101d4a32" containerName="revoke-keys" Oct 30 06:54:07.267000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.266944 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" containerName="istio-proxy" Oct 30 06:54:07.267000 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.266949 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" containerName="istio-proxy" Oct 30 06:54:07.267238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.267071 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="53abaaf4-3338-48a9-8834-a24490e47beb" containerName="revoke-keys" Oct 30 06:54:07.267238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.267081 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8112df-d5d4-41e4-adb2-560640e069b9" containerName="main" Oct 30 06:54:07.267238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.267088 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="39709b3b-ac31-48ce-8399-6c59101d4a32" containerName="revoke-keys" Oct 30 06:54:07.267238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.267098 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24b04da-c878-487b-8805-18e48cfc7f37" containerName="istio-proxy" Oct 30 06:54:07.272497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.272476 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:07.298336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.295850 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:54:07.298336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.295936 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:54:07.298336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.295965 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:54:07.298336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.296042 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:54:07.310336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.300769 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96"] Oct 30 06:54:07.389255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.389211 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbq5x\" (UniqueName: \"kubernetes.io/projected/bd922770-1bc1-4060-aa5c-b069b7dc69f4-kube-api-access-pbq5x\") pod \"maas-api-revoke-keys-e2e-inf-b-b66aad-82j96\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:07.389478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.389282 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b66aad-82j96\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:07.389574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.389545 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b66aad-82j96\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:07.490884 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.490840 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b66aad-82j96\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:07.491148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.491126 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbq5x\" (UniqueName: \"kubernetes.io/projected/bd922770-1bc1-4060-aa5c-b069b7dc69f4-kube-api-access-pbq5x\") pod \"maas-api-revoke-keys-e2e-inf-b-b66aad-82j96\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:07.491232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.491202 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b66aad-82j96\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:07.492136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.492109 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b66aad-82j96\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:07.494037 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.494010 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b66aad-82j96\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:07.514727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.514697 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbq5x\" (UniqueName: \"kubernetes.io/projected/bd922770-1bc1-4060-aa5c-b069b7dc69f4-kube-api-access-pbq5x\") pod \"maas-api-revoke-keys-e2e-inf-b-b66aad-82j96\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:07.595540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.595452 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:07.918742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:07.918712 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96"] Oct 30 06:54:07.919321 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:54:07.919283 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd922770_1bc1_4060_aa5c_b069b7dc69f4.slice/crio-b23a45d21aa7ee81b0db0dc5f4d429a35f6f827e28a56d495499578946418650 WatchSource:0}: Error finding container b23a45d21aa7ee81b0db0dc5f4d429a35f6f827e28a56d495499578946418650: Status 404 returned error can't find the container with id b23a45d21aa7ee81b0db0dc5f4d429a35f6f827e28a56d495499578946418650 Oct 30 06:54:08.166096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:08.166001 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" event={"ID":"bd922770-1bc1-4060-aa5c-b069b7dc69f4","Type":"ContainerStarted","Data":"b23a45d21aa7ee81b0db0dc5f4d429a35f6f827e28a56d495499578946418650"} Oct 30 06:54:08.269817 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:54:08.269779 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:54:09.180155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:09.180121 5001 generic.go:356] "Generic (PLEG): container finished" podID="bd922770-1bc1-4060-aa5c-b069b7dc69f4" containerID="11905891ff322cb2d017bafb425aacf87cd8a3ec682b0de930bf80df10e752ec" exitCode=0 Oct 30 06:54:09.180434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:09.180221 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" event={"ID":"bd922770-1bc1-4060-aa5c-b069b7dc69f4","Type":"ContainerDied","Data":"11905891ff322cb2d017bafb425aacf87cd8a3ec682b0de930bf80df10e752ec"} Oct 30 06:54:10.488425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:10.488391 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:10.650047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:10.650013 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-openshift-ca\") pod \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " Oct 30 06:54:10.650262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:10.650112 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd922770-1bc1-4060-aa5c-b069b7dc69f4-kube-api-access-pbq5x\" (UniqueName: \"kubernetes.io/projected/bd922770-1bc1-4060-aa5c-b069b7dc69f4-kube-api-access-pbq5x\") pod \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " Oct 30 06:54:10.650262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:10.650193 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-xks-ca\") pod \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\" (UID: \"bd922770-1bc1-4060-aa5c-b069b7dc69f4\") " Oct 30 06:54:10.650487 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:10.650458 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-openshift-ca" pod "bd922770-1bc1-4060-aa5c-b069b7dc69f4" (UID: "bd922770-1bc1-4060-aa5c-b069b7dc69f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:10.650858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:10.650830 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:10.652722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:10.652688 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-xks-ca" pod "bd922770-1bc1-4060-aa5c-b069b7dc69f4" (UID: "bd922770-1bc1-4060-aa5c-b069b7dc69f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:54:10.652877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:10.652851 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd922770-1bc1-4060-aa5c-b069b7dc69f4-kube-api-access-pbq5x" pod "bd922770-1bc1-4060-aa5c-b069b7dc69f4" (UID: "bd922770-1bc1-4060-aa5c-b069b7dc69f4"). InnerVolumeSpecName "kube-api-access-pbq5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:10.752534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:10.752440 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbq5x\" (UniqueName: \"kubernetes.io/projected/bd922770-1bc1-4060-aa5c-b069b7dc69f4-kube-api-access-pbq5x\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:10.752534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:10.752472 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd922770-1bc1-4060-aa5c-b069b7dc69f4-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:11.204522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:11.204495 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" Oct 30 06:54:11.204726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:11.204542 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96" event={"ID":"bd922770-1bc1-4060-aa5c-b069b7dc69f4","Type":"ContainerDied","Data":"b23a45d21aa7ee81b0db0dc5f4d429a35f6f827e28a56d495499578946418650"} Oct 30 06:54:11.204726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:11.204600 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b23a45d21aa7ee81b0db0dc5f4d429a35f6f827e28a56d495499578946418650" Oct 30 06:54:11.958802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:11.958770 5001 scope.go:122] "RemoveContainer" containerID="01d82343070266aac2fb4d59b57ee49ac332d3b684bf79df38a1d80e1af49c74" Oct 30 06:54:11.974931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:11.974909 5001 scope.go:122] "RemoveContainer" containerID="d63be31702b9f93109a5dc6d4dee266627f616963b427ed9c356b34a951cf7af" Oct 30 06:54:11.989785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:11.989759 5001 scope.go:122] "RemoveContainer" containerID="cca24a341b0f60df1ad73f690e85392ac0c328cbee9bd14febc715cd4fcc7e52" Oct 30 06:54:12.008815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:12.008788 5001 scope.go:122] "RemoveContainer" containerID="2124aa42c74ce89ae4667a32c2ecee897ba8920998311c0bbb2dcc98fe2b8f8d" Oct 30 06:54:17.740409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:17.740372 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj"] Oct 30 06:54:17.774111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:17.774073 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb"] Oct 30 06:54:18.087623 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:18.087539 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:54:18.749229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:18.749191 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p"] Oct 30 06:54:18.749778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:18.749431 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" containerID="cri-o://372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3" gracePeriod=30 Oct 30 06:54:22.740698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:22.740650 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" podUID="c94e7069-6896-4127-b907-0a72f7b74da3" containerName="payload-processing" containerID="cri-o://ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f" gracePeriod=25 Oct 30 06:54:22.778578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:22.778497 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" podUID="75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6" containerName="payload-pre-processing" containerID="cri-o://665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c" gracePeriod=25 Oct 30 06:54:24.081320 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.081273 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:54:24.123818 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.123783 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-envoy\") pod \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " Oct 30 06:54:24.124012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.123858 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-socket\") pod \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " Oct 30 06:54:24.124012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.123903 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-podinfo\") pod \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " Oct 30 06:54:24.124012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.123955 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-data\") pod \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " Oct 30 06:54:24.124221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.124081 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-kube-api-access-vtphh\" (UniqueName: \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-kube-api-access-vtphh\") pod \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " Oct 30 06:54:24.124221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.124159 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-token\" (UniqueName: \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-token\") pod \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " Oct 30 06:54:24.124221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.124210 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-credential-socket\") pod \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " Oct 30 06:54:24.124436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.124322 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-certs\") pod \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " Oct 30 06:54:24.124436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.124410 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istiod-ca-cert\") pod \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\" (UID: \"1fc4b90d-d07f-4f35-91ea-41dca25a573d\") " Oct 30 06:54:24.125033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.124994 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-socket" pod "1fc4b90d-d07f-4f35-91ea-41dca25a573d" (UID: "1fc4b90d-d07f-4f35-91ea-41dca25a573d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.125359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.125334 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.125466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.125374 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istiod-ca-cert" pod "1fc4b90d-d07f-4f35-91ea-41dca25a573d" (UID: "1fc4b90d-d07f-4f35-91ea-41dca25a573d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:24.125567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.125536 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-data" pod "1fc4b90d-d07f-4f35-91ea-41dca25a573d" (UID: "1fc4b90d-d07f-4f35-91ea-41dca25a573d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.125707 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.125566 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-certs" pod "1fc4b90d-d07f-4f35-91ea-41dca25a573d" (UID: "1fc4b90d-d07f-4f35-91ea-41dca25a573d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.125849 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.125825 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-credential-socket" pod "1fc4b90d-d07f-4f35-91ea-41dca25a573d" (UID: "1fc4b90d-d07f-4f35-91ea-41dca25a573d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.126924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.126902 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-envoy" pod "1fc4b90d-d07f-4f35-91ea-41dca25a573d" (UID: "1fc4b90d-d07f-4f35-91ea-41dca25a573d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.127618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.127585 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-token" pod "1fc4b90d-d07f-4f35-91ea-41dca25a573d" (UID: "1fc4b90d-d07f-4f35-91ea-41dca25a573d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:24.128148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.128124 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-kube-api-access-vtphh" pod "1fc4b90d-d07f-4f35-91ea-41dca25a573d" (UID: "1fc4b90d-d07f-4f35-91ea-41dca25a573d"). InnerVolumeSpecName "kube-api-access-vtphh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:24.128355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.128295 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-podinfo" pod "1fc4b90d-d07f-4f35-91ea-41dca25a573d" (UID: "1fc4b90d-d07f-4f35-91ea-41dca25a573d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:54:24.227381 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.227296 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.227577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.227559 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.227686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.227580 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.227686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.227594 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtphh\" (UniqueName: \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-kube-api-access-vtphh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.227686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.227608 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.227686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.227621 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.227686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.227633 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc4b90d-d07f-4f35-91ea-41dca25a573d-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.227686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.227645 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc4b90d-d07f-4f35-91ea-41dca25a573d-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.429987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.429945 5001 generic.go:356] "Generic (PLEG): container finished" podID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerID="372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3" exitCode=0 Oct 30 06:54:24.430212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.430102 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" event={"ID":"1fc4b90d-d07f-4f35-91ea-41dca25a573d","Type":"ContainerDied","Data":"372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3"} Oct 30 06:54:24.430212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.430141 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" event={"ID":"1fc4b90d-d07f-4f35-91ea-41dca25a573d","Type":"ContainerDied","Data":"75daeba016f0387c788cb0d1135c849f3bdd7f55159d73bc6508014551d4d89f"} Oct 30 06:54:24.430212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.430169 5001 scope.go:122] "RemoveContainer" containerID="372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3" Oct 30 06:54:24.430461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.430337 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p" Oct 30 06:54:24.445467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.445446 5001 scope.go:122] "RemoveContainer" containerID="372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3" Oct 30 06:54:24.445825 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:54:24.445799 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3\": container with ID starting with 372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3 not found: ID does not exist" containerID="372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3" Oct 30 06:54:24.445923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.445834 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3"} err="failed to get container status \"372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3\": rpc error: code = NotFound desc = could not find container \"372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3\": container with ID starting with 372658082a477ec5023d204cf010192270568c7d2ba4763d18a43f5320d182c3 not found: ID does not exist" Oct 30 06:54:24.498935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.498903 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p"] Oct 30 06:54:24.507211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:24.507175 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4fh4p"] Oct 30 06:54:26.120665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:26.120632 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" path="/var/lib/kubelet/pods/1fc4b90d-d07f-4f35-91ea-41dca25a573d/volumes" Oct 30 06:54:32.145122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:32.145073 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" podUID="75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:54:32.145559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:32.145087 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" podUID="c94e7069-6896-4127-b907-0a72f7b74da3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:54:38.272441 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:54:38.272385 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:54:42.145224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:42.145187 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" podUID="c94e7069-6896-4127-b907-0a72f7b74da3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:54:42.145701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:42.145227 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" podUID="75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:54:43.110485 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.110457 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:54:43.127091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.127062 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94e7069-6896-4127-b907-0a72f7b74da3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94e7069-6896-4127-b907-0a72f7b74da3-plugins-config-volume\") pod \"c94e7069-6896-4127-b907-0a72f7b74da3\" (UID: \"c94e7069-6896-4127-b907-0a72f7b74da3\") " Oct 30 06:54:43.127216 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.127134 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94e7069-6896-4127-b907-0a72f7b74da3-kube-api-access-zqksp\" (UniqueName: \"kubernetes.io/projected/c94e7069-6896-4127-b907-0a72f7b74da3-kube-api-access-zqksp\") pod \"c94e7069-6896-4127-b907-0a72f7b74da3\" (UID: \"c94e7069-6896-4127-b907-0a72f7b74da3\") " Oct 30 06:54:43.127509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.127484 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94e7069-6896-4127-b907-0a72f7b74da3-plugins-config-volume" pod "c94e7069-6896-4127-b907-0a72f7b74da3" (UID: "c94e7069-6896-4127-b907-0a72f7b74da3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:43.128060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.128038 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94e7069-6896-4127-b907-0a72f7b74da3-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:43.129520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.129495 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94e7069-6896-4127-b907-0a72f7b74da3-kube-api-access-zqksp" pod "c94e7069-6896-4127-b907-0a72f7b74da3" (UID: "c94e7069-6896-4127-b907-0a72f7b74da3"). InnerVolumeSpecName "kube-api-access-zqksp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:43.136589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.136570 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:54:43.229004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.228975 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-kube-api-access-tm4sm\" (UniqueName: \"kubernetes.io/projected/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-kube-api-access-tm4sm\") pod \"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6\" (UID: \"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6\") " Oct 30 06:54:43.229805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.229031 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-plugins-config-volume\") pod \"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6\" (UID: \"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6\") " Oct 30 06:54:43.230190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.230124 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqksp\" (UniqueName: \"kubernetes.io/projected/c94e7069-6896-4127-b907-0a72f7b74da3-kube-api-access-zqksp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:43.230394 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.230273 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-plugins-config-volume" pod "75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6" (UID: "75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:43.232336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.232287 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-kube-api-access-tm4sm" pod "75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6" (UID: "75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6"). InnerVolumeSpecName "kube-api-access-tm4sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:43.331529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.331442 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm4sm\" (UniqueName: \"kubernetes.io/projected/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-kube-api-access-tm4sm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:43.331529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.331470 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:54:43.648520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.648487 5001 generic.go:356] "Generic (PLEG): container finished" podID="c94e7069-6896-4127-b907-0a72f7b74da3" containerID="ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f" exitCode=0 Oct 30 06:54:43.648737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.648536 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" event={"ID":"c94e7069-6896-4127-b907-0a72f7b74da3","Type":"ContainerDied","Data":"ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f"} Oct 30 06:54:43.648737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.648554 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" Oct 30 06:54:43.648737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.648599 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj" event={"ID":"c94e7069-6896-4127-b907-0a72f7b74da3","Type":"ContainerDied","Data":"3f87d0b071badba583d2613a48ad29d2fc0f78c163a218855f7128653697d2cf"} Oct 30 06:54:43.648737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.648629 5001 scope.go:122] "RemoveContainer" containerID="ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f" Oct 30 06:54:43.650459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.650436 5001 generic.go:356] "Generic (PLEG): container finished" podID="75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6" containerID="665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c" exitCode=0 Oct 30 06:54:43.650659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.650497 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" Oct 30 06:54:43.650790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.650519 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" event={"ID":"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6","Type":"ContainerDied","Data":"665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c"} Oct 30 06:54:43.650790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.650769 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb" event={"ID":"75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6","Type":"ContainerDied","Data":"92c88808847ad42e3d67c1a883951e909f7bdd1581b12d94bb35c488d6aee30c"} Oct 30 06:54:43.663923 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.663901 5001 scope.go:122] "RemoveContainer" containerID="ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f" Oct 30 06:54:43.664250 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:54:43.664230 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f\": container with ID starting with ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f not found: ID does not exist" containerID="ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f" Oct 30 06:54:43.664336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.664260 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f"} err="failed to get container status \"ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f\": rpc error: code = NotFound desc = could not find container \"ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f\": container with ID starting with ec752a8c8e6be6488cf658be98f590d232b3c61ff38e57cfd1c3b045dc2b1a9f not found: ID does not exist" Oct 30 06:54:43.664336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.664281 5001 scope.go:122] "RemoveContainer" containerID="665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c" Oct 30 06:54:43.678614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.678590 5001 scope.go:122] "RemoveContainer" containerID="665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c" Oct 30 06:54:43.678913 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:54:43.678888 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c\": container with ID starting with 665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c not found: ID does not exist" containerID="665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c" Oct 30 06:54:43.679016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.678925 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c"} err="failed to get container status \"665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c\": rpc error: code = NotFound desc = could not find container \"665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c\": container with ID starting with 665189bf4b2636a3b0f41d777e95a8fb5e741ad6963d722f38ae6a2a915c557c not found: ID does not exist" Oct 30 06:54:43.703066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.703034 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb"] Oct 30 06:54:43.714101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.714071 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b66aad-7994fbc7cf-mh9zb"] Oct 30 06:54:43.741968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.741937 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj"] Oct 30 06:54:43.748893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:43.748867 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b66aad-6b96d78f98-zlkbj"] Oct 30 06:54:44.124183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:44.124144 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6" path="/var/lib/kubelet/pods/75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6/volumes" Oct 30 06:54:44.124987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:44.124959 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94e7069-6896-4127-b907-0a72f7b74da3" path="/var/lib/kubelet/pods/c94e7069-6896-4127-b907-0a72f7b74da3/volumes" Oct 30 06:54:50.123376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:50.123291 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr"] Oct 30 06:54:50.128489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:50.128456 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-eaee88-65gmr"] Oct 30 06:54:52.120414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:52.120377 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebff91a-90df-4838-8b6d-3a0902b69337" path="/var/lib/kubelet/pods/0ebff91a-90df-4838-8b6d-3a0902b69337/volumes" Oct 30 06:54:56.051691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:56.051651 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96"] Oct 30 06:54:56.073377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:56.064800 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b66aad-82j96"] Oct 30 06:54:56.126505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:54:56.126472 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd922770-1bc1-4060-aa5c-b069b7dc69f4" path="/var/lib/kubelet/pods/bd922770-1bc1-4060-aa5c-b069b7dc69f4/volumes" Oct 30 06:55:00.901158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:00.901116 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:55:04.325803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.325457 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v"] Oct 30 06:55:04.325803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.325711 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" containerID="cri-o://c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c" gracePeriod=30 Oct 30 06:55:04.988047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.988000 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t"] Oct 30 06:55:04.989358 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989333 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c94e7069-6896-4127-b907-0a72f7b74da3" containerName="payload-processing" Oct 30 06:55:04.989453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989359 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94e7069-6896-4127-b907-0a72f7b74da3" containerName="payload-processing" Oct 30 06:55:04.989453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989382 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" Oct 30 06:55:04.989453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989389 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" Oct 30 06:55:04.989453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989422 5001 cpu_manager.go:414] "containerMap: removing container" podUID="bd922770-1bc1-4060-aa5c-b069b7dc69f4" containerName="revoke-keys" Oct 30 06:55:04.989453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989430 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd922770-1bc1-4060-aa5c-b069b7dc69f4" containerName="revoke-keys" Oct 30 06:55:04.989636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989496 5001 cpu_manager.go:414] "containerMap: removing container" podUID="75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6" containerName="payload-pre-processing" Oct 30 06:55:04.989636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989504 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6" containerName="payload-pre-processing" Oct 30 06:55:04.989716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989708 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d30c14-d3cf-46b1-8dd8-2f451f1bb8d6" containerName="payload-pre-processing" Oct 30 06:55:04.989769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989728 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94e7069-6896-4127-b907-0a72f7b74da3" containerName="payload-processing" Oct 30 06:55:04.989769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989740 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc4b90d-d07f-4f35-91ea-41dca25a573d" containerName="istio-proxy" Oct 30 06:55:04.989769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.989754 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd922770-1bc1-4060-aa5c-b069b7dc69f4" containerName="revoke-keys" Oct 30 06:55:04.996387 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:04.996299 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:05.010712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.010669 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:55:05.010878 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.010733 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:55:05.011369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.011327 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t"] Oct 30 06:55:05.041821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.041764 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:55:05.042211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.042190 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:55:05.097421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.097374 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:05.097618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.097582 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:05.097713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.097697 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn9qq\" (UniqueName: \"kubernetes.io/projected/792b027a-6f45-4c89-b95b-c2a380c2c6e2-kube-api-access-xn9qq\") pod \"maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:05.200119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.199559 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:05.200589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.200560 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:05.200722 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.200689 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn9qq\" (UniqueName: \"kubernetes.io/projected/792b027a-6f45-4c89-b95b-c2a380c2c6e2-kube-api-access-xn9qq\") pod \"maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:05.201489 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.201455 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:05.202662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.202634 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:05.223751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.223724 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn9qq\" (UniqueName: \"kubernetes.io/projected/792b027a-6f45-4c89-b95b-c2a380c2c6e2-kube-api-access-xn9qq\") pod \"maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:05.316663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.316574 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:05.640242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:05.639702 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t"] Oct 30 06:55:06.059456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:06.059422 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" event={"ID":"792b027a-6f45-4c89-b95b-c2a380c2c6e2","Type":"ContainerStarted","Data":"531416ff7e03463ba21145a48122a876e0777559b5845b9060a01f215ba87751"} Oct 30 06:55:07.089415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:07.089278 5001 generic.go:356] "Generic (PLEG): container finished" podID="792b027a-6f45-4c89-b95b-c2a380c2c6e2" containerID="ac2a932c98783917e2cd731f365d9254a9968e28efb4b64a7b95522ba2c9aa39" exitCode=0 Oct 30 06:55:07.089415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:07.089359 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" event={"ID":"792b027a-6f45-4c89-b95b-c2a380c2c6e2","Type":"ContainerDied","Data":"ac2a932c98783917e2cd731f365d9254a9968e28efb4b64a7b95522ba2c9aa39"} Oct 30 06:55:08.297702 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:55:08.297661 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:55:08.303006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.302971 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:08.428163 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.428125 5001 patch_prober.go:29] interesting pod/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:55:08.428356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.428186 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:55:08.466073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.466037 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792b027a-6f45-4c89-b95b-c2a380c2c6e2-kube-api-access-xn9qq\" (UniqueName: \"kubernetes.io/projected/792b027a-6f45-4c89-b95b-c2a380c2c6e2-kube-api-access-xn9qq\") pod \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " Oct 30 06:55:08.466283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.466163 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-openshift-ca\") pod \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " Oct 30 06:55:08.466283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.466188 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-xks-ca\") pod \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\" (UID: \"792b027a-6f45-4c89-b95b-c2a380c2c6e2\") " Oct 30 06:55:08.466633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.466600 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-openshift-ca" pod "792b027a-6f45-4c89-b95b-c2a380c2c6e2" (UID: "792b027a-6f45-4c89-b95b-c2a380c2c6e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:55:08.468656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.468624 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792b027a-6f45-4c89-b95b-c2a380c2c6e2-kube-api-access-xn9qq" pod "792b027a-6f45-4c89-b95b-c2a380c2c6e2" (UID: "792b027a-6f45-4c89-b95b-c2a380c2c6e2"). InnerVolumeSpecName "kube-api-access-xn9qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:08.468656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.468627 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-xks-ca" pod "792b027a-6f45-4c89-b95b-c2a380c2c6e2" (UID: "792b027a-6f45-4c89-b95b-c2a380c2c6e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:55:08.568641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.568599 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn9qq\" (UniqueName: \"kubernetes.io/projected/792b027a-6f45-4c89-b95b-c2a380c2c6e2-kube-api-access-xn9qq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:08.568641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.568639 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:08.568888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:08.568654 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792b027a-6f45-4c89-b95b-c2a380c2c6e2-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:09.109237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.109202 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" Oct 30 06:55:09.109237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.109231 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t" event={"ID":"792b027a-6f45-4c89-b95b-c2a380c2c6e2","Type":"ContainerDied","Data":"531416ff7e03463ba21145a48122a876e0777559b5845b9060a01f215ba87751"} Oct 30 06:55:09.109536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.109274 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="531416ff7e03463ba21145a48122a876e0777559b5845b9060a01f215ba87751" Oct 30 06:55:09.642990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.642919 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:55:09.792176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792128 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-data\") pod \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " Oct 30 06:55:09.792417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792197 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-socket\") pod \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " Oct 30 06:55:09.792417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792268 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istiod-ca-cert\") pod \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " Oct 30 06:55:09.792417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792366 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-podinfo\") pod \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " Oct 30 06:55:09.792604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792552 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-certs\") pod \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " Oct 30 06:55:09.792604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792553 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-data" pod "3edcda5b-e672-43ae-9fdf-10b0a644d0ae" (UID: "3edcda5b-e672-43ae-9fdf-10b0a644d0ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:09.792604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792568 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-socket" pod "3edcda5b-e672-43ae-9fdf-10b0a644d0ae" (UID: "3edcda5b-e672-43ae-9fdf-10b0a644d0ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:09.792782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792689 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-token\" (UniqueName: \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-token\") pod \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " Oct 30 06:55:09.792782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792740 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-credential-socket\") pod \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " Oct 30 06:55:09.792893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792830 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-certs" pod "3edcda5b-e672-43ae-9fdf-10b0a644d0ae" (UID: "3edcda5b-e672-43ae-9fdf-10b0a644d0ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:09.792893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792847 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-envoy\") pod \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " Oct 30 06:55:09.793009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792944 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-kube-api-access-fq6mb\" (UniqueName: \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-kube-api-access-fq6mb\") pod \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\" (UID: \"3edcda5b-e672-43ae-9fdf-10b0a644d0ae\") " Oct 30 06:55:09.793009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.792948 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istiod-ca-cert" pod "3edcda5b-e672-43ae-9fdf-10b0a644d0ae" (UID: "3edcda5b-e672-43ae-9fdf-10b0a644d0ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:55:09.793337 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.793230 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-credential-socket" pod "3edcda5b-e672-43ae-9fdf-10b0a644d0ae" (UID: "3edcda5b-e672-43ae-9fdf-10b0a644d0ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:09.793787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.793762 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:09.793963 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.793946 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:09.794072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.794059 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:09.794175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.794163 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:09.794267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.794256 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:09.796105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.796075 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-podinfo" pod "3edcda5b-e672-43ae-9fdf-10b0a644d0ae" (UID: "3edcda5b-e672-43ae-9fdf-10b0a644d0ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:55:09.796235 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.796130 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-token" pod "3edcda5b-e672-43ae-9fdf-10b0a644d0ae" (UID: "3edcda5b-e672-43ae-9fdf-10b0a644d0ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:09.796611 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.796589 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-kube-api-access-fq6mb" pod "3edcda5b-e672-43ae-9fdf-10b0a644d0ae" (UID: "3edcda5b-e672-43ae-9fdf-10b0a644d0ae"). InnerVolumeSpecName "kube-api-access-fq6mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:09.798065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.798043 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-envoy" pod "3edcda5b-e672-43ae-9fdf-10b0a644d0ae" (UID: "3edcda5b-e672-43ae-9fdf-10b0a644d0ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:09.895811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.895711 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:09.895811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.895745 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:09.895811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.895760 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6mb\" (UniqueName: \"kubernetes.io/projected/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-kube-api-access-fq6mb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:09.895811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:09.895771 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edcda5b-e672-43ae-9fdf-10b0a644d0ae-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:10.122843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.122809 5001 generic.go:356] "Generic (PLEG): container finished" podID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerID="c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c" exitCode=0 Oct 30 06:55:10.123018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.122914 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" Oct 30 06:55:10.123277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.123256 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" event={"ID":"3edcda5b-e672-43ae-9fdf-10b0a644d0ae","Type":"ContainerDied","Data":"c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c"} Oct 30 06:55:10.123352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.123323 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v" event={"ID":"3edcda5b-e672-43ae-9fdf-10b0a644d0ae","Type":"ContainerDied","Data":"a702796ef4ee99fd0ff4846f0c459e2967bafd6a4c4864015ccf19e7f11be073"} Oct 30 06:55:10.123410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.123357 5001 scope.go:122] "RemoveContainer" containerID="c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c" Oct 30 06:55:10.138682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.138660 5001 scope.go:122] "RemoveContainer" containerID="60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a" Oct 30 06:55:10.150613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.150581 5001 scope.go:122] "RemoveContainer" containerID="c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c" Oct 30 06:55:10.150951 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:55:10.150924 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c\": container with ID starting with c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c not found: ID does not exist" containerID="c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c" Oct 30 06:55:10.151013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.150963 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c"} err="failed to get container status \"c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c\": rpc error: code = NotFound desc = could not find container \"c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c\": container with ID starting with c395c3bea014e776da59f164f9c48a781c45c8225e0fecf8e931cb248164ca8c not found: ID does not exist" Oct 30 06:55:10.151013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.150992 5001 scope.go:122] "RemoveContainer" containerID="60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a" Oct 30 06:55:10.151261 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:55:10.151243 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a\": container with ID starting with 60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a not found: ID does not exist" containerID="60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a" Oct 30 06:55:10.151662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.151267 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a"} err="failed to get container status \"60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a\": rpc error: code = NotFound desc = could not find container \"60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a\": container with ID starting with 60ef6fb72daf403978f5d0a3c10d6a63c647f979f15399eb58e12cbafaab903a not found: ID does not exist" Oct 30 06:55:10.179709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.179674 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v"] Oct 30 06:55:10.191131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:10.191098 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b66aad-openshift-default-5cfcc749d9-tdh9v"] Oct 30 06:55:12.122297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:12.122263 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" path="/var/lib/kubelet/pods/3edcda5b-e672-43ae-9fdf-10b0a644d0ae/volumes" Oct 30 06:55:15.462895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:15.462858 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm"] Oct 30 06:55:20.463361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:20.463247 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" podUID="2e9c2db0-e1f2-4817-8bb0-90062efa45e1" containerName="payload-pre-processing" containerID="cri-o://13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b" gracePeriod=25 Oct 30 06:55:23.114857 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:23.114812 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" podUID="2e9c2db0-e1f2-4817-8bb0-90062efa45e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:55:33.114848 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:33.114810 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" podUID="2e9c2db0-e1f2-4817-8bb0-90062efa45e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:55:38.270316 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:55:38.270260 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:55:40.797083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:40.797056 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:55:40.999587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:40.996453 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-plugins-config-volume\") pod \"2e9c2db0-e1f2-4817-8bb0-90062efa45e1\" (UID: \"2e9c2db0-e1f2-4817-8bb0-90062efa45e1\") " Oct 30 06:55:40.999587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:40.996675 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-kube-api-access-kqpzt\" (UniqueName: \"kubernetes.io/projected/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-kube-api-access-kqpzt\") pod \"2e9c2db0-e1f2-4817-8bb0-90062efa45e1\" (UID: \"2e9c2db0-e1f2-4817-8bb0-90062efa45e1\") " Oct 30 06:55:40.999587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:40.997417 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-plugins-config-volume" pod "2e9c2db0-e1f2-4817-8bb0-90062efa45e1" (UID: "2e9c2db0-e1f2-4817-8bb0-90062efa45e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:55:41.009263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.009223 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-kube-api-access-kqpzt" pod "2e9c2db0-e1f2-4817-8bb0-90062efa45e1" (UID: "2e9c2db0-e1f2-4817-8bb0-90062efa45e1"). InnerVolumeSpecName "kube-api-access-kqpzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:41.098837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.098794 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqpzt\" (UniqueName: \"kubernetes.io/projected/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-kube-api-access-kqpzt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:41.098837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.098835 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9c2db0-e1f2-4817-8bb0-90062efa45e1-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:41.532493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.532457 5001 generic.go:356] "Generic (PLEG): container finished" podID="2e9c2db0-e1f2-4817-8bb0-90062efa45e1" containerID="13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b" exitCode=0 Oct 30 06:55:41.532717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.532519 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" Oct 30 06:55:41.532717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.532549 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" event={"ID":"2e9c2db0-e1f2-4817-8bb0-90062efa45e1","Type":"ContainerDied","Data":"13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b"} Oct 30 06:55:41.532717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.532618 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm" event={"ID":"2e9c2db0-e1f2-4817-8bb0-90062efa45e1","Type":"ContainerDied","Data":"2e319cd699b82a2f4a0bd412fd5572e547fa382afd3448d5d0d6ea1b1ecdad79"} Oct 30 06:55:41.532717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.532649 5001 scope.go:122] "RemoveContainer" containerID="13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b" Oct 30 06:55:41.546938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.546896 5001 scope.go:122] "RemoveContainer" containerID="13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b" Oct 30 06:55:41.547265 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:55:41.547233 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b\": container with ID starting with 13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b not found: ID does not exist" containerID="13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b" Oct 30 06:55:41.547359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.547282 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b"} err="failed to get container status \"13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b\": rpc error: code = NotFound desc = could not find container \"13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b\": container with ID starting with 13bf32293d86e57262bae377a9c58d694336d3f945eb0c183834e2b83e07d04b not found: ID does not exist" Oct 30 06:55:41.586939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.586898 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm"] Oct 30 06:55:41.594153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:41.594125 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-88acce-696f848bc9-qxfgm"] Oct 30 06:55:42.122630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:42.122594 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e9c2db0-e1f2-4817-8bb0-90062efa45e1" path="/var/lib/kubelet/pods/2e9c2db0-e1f2-4817-8bb0-90062efa45e1/volumes" Oct 30 06:55:43.178642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:43.178607 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t"] Oct 30 06:55:43.183170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:43.183135 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-88acce-tnh4t"] Oct 30 06:55:44.121238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:44.121200 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792b027a-6f45-4c89-b95b-c2a380c2c6e2" path="/var/lib/kubelet/pods/792b027a-6f45-4c89-b95b-c2a380c2c6e2/volumes" Oct 30 06:55:50.412710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:50.412679 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq"] Oct 30 06:55:50.413463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:50.412924 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" containerID="cri-o://4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c" gracePeriod=30 Oct 30 06:55:51.061483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.061437 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv"] Oct 30 06:55:51.062695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.062665 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" Oct 30 06:55:51.062695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.062693 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" Oct 30 06:55:51.062912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.062713 5001 cpu_manager.go:414] "containerMap: removing container" podUID="792b027a-6f45-4c89-b95b-c2a380c2c6e2" containerName="revoke-keys" Oct 30 06:55:51.062912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.062720 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792b027a-6f45-4c89-b95b-c2a380c2c6e2" containerName="revoke-keys" Oct 30 06:55:51.062912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.062769 5001 cpu_manager.go:414] "containerMap: removing container" podUID="2e9c2db0-e1f2-4817-8bb0-90062efa45e1" containerName="payload-pre-processing" Oct 30 06:55:51.062912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.062777 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e9c2db0-e1f2-4817-8bb0-90062efa45e1" containerName="payload-pre-processing" Oct 30 06:55:51.063107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.062991 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" Oct 30 06:55:51.063107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.063012 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e9c2db0-e1f2-4817-8bb0-90062efa45e1" containerName="payload-pre-processing" Oct 30 06:55:51.063107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.063035 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="792b027a-6f45-4c89-b95b-c2a380c2c6e2" containerName="revoke-keys" Oct 30 06:55:51.069456 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.069426 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:51.085355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.083548 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:55:51.085355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.085173 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv"] Oct 30 06:55:51.089426 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.089405 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:55:51.089577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.089434 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:55:51.089577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.089441 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:55:51.232833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.232785 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnpc5\" (UniqueName: \"kubernetes.io/projected/822010df-d574-4cab-b041-e4f2d4fda3a2-kube-api-access-vnpc5\") pod \"maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:51.233019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.232861 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:51.233162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.233124 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:51.334809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.334715 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:51.334995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.334903 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnpc5\" (UniqueName: \"kubernetes.io/projected/822010df-d574-4cab-b041-e4f2d4fda3a2-kube-api-access-vnpc5\") pod \"maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:51.334995 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.334959 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:51.335541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.335515 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:51.339704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.337655 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:51.359507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.359468 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnpc5\" (UniqueName: \"kubernetes.io/projected/822010df-d574-4cab-b041-e4f2d4fda3a2-kube-api-access-vnpc5\") pod \"maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:51.389440 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.389402 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:51.709838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:51.709806 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv"] Oct 30 06:55:51.714280 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:55:51.714249 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822010df_d574_4cab_b041_e4f2d4fda3a2.slice/crio-69768757bdee7e6dd65d353142dfadcd4670248c28ded07b3dee0f4fecc4e48e WatchSource:0}: Error finding container 69768757bdee7e6dd65d353142dfadcd4670248c28ded07b3dee0f4fecc4e48e: Status 404 returned error can't find the container with id 69768757bdee7e6dd65d353142dfadcd4670248c28ded07b3dee0f4fecc4e48e Oct 30 06:55:52.720014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:52.719971 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" event={"ID":"822010df-d574-4cab-b041-e4f2d4fda3a2","Type":"ContainerStarted","Data":"9a9c7a5a37ce2b163516b7ce18cf467b375928480dd5b03a6bb244e5224c00dd"} Oct 30 06:55:52.720543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:52.720078 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" event={"ID":"822010df-d574-4cab-b041-e4f2d4fda3a2","Type":"ContainerStarted","Data":"69768757bdee7e6dd65d353142dfadcd4670248c28ded07b3dee0f4fecc4e48e"} Oct 30 06:55:52.762425 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:52.762325 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" podStartSLOduration=1.762294274 podStartE2EDuration="1.762294274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:55:52.761067842 +0000 UTC m=+3766.757024852" watchObservedRunningTime="2026-10-30 06:55:52.762294274 +0000 UTC m=+3766.758251284" Oct 30 06:55:53.689006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:53.688972 5001 patch_prober.go:29] interesting pod/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:55:53.689229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:53.689033 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:55:53.730175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:53.730140 5001 generic.go:356] "Generic (PLEG): container finished" podID="822010df-d574-4cab-b041-e4f2d4fda3a2" containerID="9a9c7a5a37ce2b163516b7ce18cf467b375928480dd5b03a6bb244e5224c00dd" exitCode=0 Oct 30 06:55:53.730649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:53.730231 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" event={"ID":"822010df-d574-4cab-b041-e4f2d4fda3a2","Type":"ContainerDied","Data":"9a9c7a5a37ce2b163516b7ce18cf467b375928480dd5b03a6bb244e5224c00dd"} Oct 30 06:55:54.944774 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:54.944744 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:55.013760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.013656 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-xks-ca\") pod \"822010df-d574-4cab-b041-e4f2d4fda3a2\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " Oct 30 06:55:55.013959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.013787 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-openshift-ca\") pod \"822010df-d574-4cab-b041-e4f2d4fda3a2\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " Oct 30 06:55:55.013959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.013885 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822010df-d574-4cab-b041-e4f2d4fda3a2-kube-api-access-vnpc5\" (UniqueName: \"kubernetes.io/projected/822010df-d574-4cab-b041-e4f2d4fda3a2-kube-api-access-vnpc5\") pod \"822010df-d574-4cab-b041-e4f2d4fda3a2\" (UID: \"822010df-d574-4cab-b041-e4f2d4fda3a2\") " Oct 30 06:55:55.014691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.014651 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-openshift-ca" pod "822010df-d574-4cab-b041-e4f2d4fda3a2" (UID: "822010df-d574-4cab-b041-e4f2d4fda3a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:55:55.016586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.016562 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-xks-ca" pod "822010df-d574-4cab-b041-e4f2d4fda3a2" (UID: "822010df-d574-4cab-b041-e4f2d4fda3a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:55:55.017548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.017504 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822010df-d574-4cab-b041-e4f2d4fda3a2-kube-api-access-vnpc5" pod "822010df-d574-4cab-b041-e4f2d4fda3a2" (UID: "822010df-d574-4cab-b041-e4f2d4fda3a2"). InnerVolumeSpecName "kube-api-access-vnpc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:55.079728 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.079681 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:55:55.115886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.115853 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.115886 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.115889 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822010df-d574-4cab-b041-e4f2d4fda3a2-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.116136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.115912 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnpc5\" (UniqueName: \"kubernetes.io/projected/822010df-d574-4cab-b041-e4f2d4fda3a2-kube-api-access-vnpc5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.680929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.680865 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:55:55.725690 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.725652 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-envoy\") pod \"981ae7ff-0c71-468e-b045-9f7735860591\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " Oct 30 06:55:55.725863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.725730 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/981ae7ff-0c71-468e-b045-9f7735860591-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/981ae7ff-0c71-468e-b045-9f7735860591-istio-podinfo\") pod \"981ae7ff-0c71-468e-b045-9f7735860591\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " Oct 30 06:55:55.725863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.725784 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-socket\") pod \"981ae7ff-0c71-468e-b045-9f7735860591\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " Oct 30 06:55:55.725863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.725820 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981ae7ff-0c71-468e-b045-9f7735860591-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/981ae7ff-0c71-468e-b045-9f7735860591-istiod-ca-cert\") pod \"981ae7ff-0c71-468e-b045-9f7735860591\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " Oct 30 06:55:55.726356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.725908 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-data\") pod \"981ae7ff-0c71-468e-b045-9f7735860591\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " Oct 30 06:55:55.726356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.725958 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-certs\") pod \"981ae7ff-0c71-468e-b045-9f7735860591\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " Oct 30 06:55:55.726356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.725994 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-istio-token\" (UniqueName: \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-istio-token\") pod \"981ae7ff-0c71-468e-b045-9f7735860591\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " Oct 30 06:55:55.726356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.726054 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-kube-api-access-5pmpv\" (UniqueName: \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-kube-api-access-5pmpv\") pod \"981ae7ff-0c71-468e-b045-9f7735860591\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " Oct 30 06:55:55.726356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.726138 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-credential-socket\") pod \"981ae7ff-0c71-468e-b045-9f7735860591\" (UID: \"981ae7ff-0c71-468e-b045-9f7735860591\") " Oct 30 06:55:55.726356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.726140 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-socket" pod "981ae7ff-0c71-468e-b045-9f7735860591" (UID: "981ae7ff-0c71-468e-b045-9f7735860591"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:55.726356 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.726201 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-data" pod "981ae7ff-0c71-468e-b045-9f7735860591" (UID: "981ae7ff-0c71-468e-b045-9f7735860591"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:55.726710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.726245 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981ae7ff-0c71-468e-b045-9f7735860591-istiod-ca-cert" pod "981ae7ff-0c71-468e-b045-9f7735860591" (UID: "981ae7ff-0c71-468e-b045-9f7735860591"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:55:55.726710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.726339 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-certs" pod "981ae7ff-0c71-468e-b045-9f7735860591" (UID: "981ae7ff-0c71-468e-b045-9f7735860591"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:55.726710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.726402 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-credential-socket" pod "981ae7ff-0c71-468e-b045-9f7735860591" (UID: "981ae7ff-0c71-468e-b045-9f7735860591"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:55.727041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.727018 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.727143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.727046 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.727143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.727060 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.727143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.727075 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.727143 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.727087 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/981ae7ff-0c71-468e-b045-9f7735860591-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.728408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.728385 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-kube-api-access-5pmpv" pod "981ae7ff-0c71-468e-b045-9f7735860591" (UID: "981ae7ff-0c71-468e-b045-9f7735860591"). InnerVolumeSpecName "kube-api-access-5pmpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:55.728408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.728386 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-envoy" pod "981ae7ff-0c71-468e-b045-9f7735860591" (UID: "981ae7ff-0c71-468e-b045-9f7735860591"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:55.728684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.728669 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/981ae7ff-0c71-468e-b045-9f7735860591-istio-podinfo" pod "981ae7ff-0c71-468e-b045-9f7735860591" (UID: "981ae7ff-0c71-468e-b045-9f7735860591"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:55:55.728782 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.728763 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-istio-token" pod "981ae7ff-0c71-468e-b045-9f7735860591" (UID: "981ae7ff-0c71-468e-b045-9f7735860591"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:55.753346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.753291 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" Oct 30 06:55:55.753346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.753328 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv" event={"ID":"822010df-d574-4cab-b041-e4f2d4fda3a2","Type":"ContainerDied","Data":"69768757bdee7e6dd65d353142dfadcd4670248c28ded07b3dee0f4fecc4e48e"} Oct 30 06:55:55.753576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.753389 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69768757bdee7e6dd65d353142dfadcd4670248c28ded07b3dee0f4fecc4e48e" Oct 30 06:55:55.755455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.755427 5001 generic.go:356] "Generic (PLEG): container finished" podID="981ae7ff-0c71-468e-b045-9f7735860591" containerID="4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c" exitCode=0 Oct 30 06:55:55.755660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.755496 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" Oct 30 06:55:55.755660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.755511 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" event={"ID":"981ae7ff-0c71-468e-b045-9f7735860591","Type":"ContainerDied","Data":"4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c"} Oct 30 06:55:55.755660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.755575 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq" event={"ID":"981ae7ff-0c71-468e-b045-9f7735860591","Type":"ContainerDied","Data":"879b54e7b41751eed888513bfb50c5cbc4c280864aab66455d55e2710e21de2f"} Oct 30 06:55:55.755660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.755606 5001 scope.go:122] "RemoveContainer" containerID="4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c" Oct 30 06:55:55.774450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.774425 5001 scope.go:122] "RemoveContainer" containerID="0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc" Oct 30 06:55:55.790041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.790014 5001 scope.go:122] "RemoveContainer" containerID="4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c" Oct 30 06:55:55.790666 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:55:55.790625 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c\": container with ID starting with 4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c not found: ID does not exist" containerID="4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c" Oct 30 06:55:55.790805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.790678 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c"} err="failed to get container status \"4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c\": rpc error: code = NotFound desc = could not find container \"4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c\": container with ID starting with 4c29b2fe23d08ef8e6e0f667942c09b00da8a4fd5017999411b7ca4d601e781c not found: ID does not exist" Oct 30 06:55:55.790805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.790704 5001 scope.go:122] "RemoveContainer" containerID="0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc" Oct 30 06:55:55.791091 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:55:55.791047 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc\": container with ID starting with 0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc not found: ID does not exist" containerID="0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc" Oct 30 06:55:55.791091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.791079 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc"} err="failed to get container status \"0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc\": rpc error: code = NotFound desc = could not find container \"0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc\": container with ID starting with 0124d8e9f883990cb9541aae9b30d11ce440863befe420b6d3b353e821ccd3bc not found: ID does not exist" Oct 30 06:55:55.828239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.828205 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.828239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.828236 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pmpv\" (UniqueName: \"kubernetes.io/projected/981ae7ff-0c71-468e-b045-9f7735860591-kube-api-access-5pmpv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.828239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.828246 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/981ae7ff-0c71-468e-b045-9f7735860591-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.828571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.828254 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/981ae7ff-0c71-468e-b045-9f7735860591-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:55:55.831548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.831523 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq"] Oct 30 06:55:55.851917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:55.851863 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-88acce-openshift-default-6955f69bdf-bzkzq"] Oct 30 06:55:56.218580 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:55:56.218537 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981ae7ff-0c71-468e-b045-9f7735860591" path="/var/lib/kubelet/pods/981ae7ff-0c71-468e-b045-9f7735860591/volumes" Oct 30 06:56:08.273214 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:56:08.273174 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:56:20.103505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.103471 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg"] Oct 30 06:56:20.104341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104322 5001 cpu_manager.go:414] "containerMap: removing container" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" Oct 30 06:56:20.104405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104343 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" Oct 30 06:56:20.104405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104395 5001 cpu_manager.go:414] "containerMap: removing container" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" Oct 30 06:56:20.104405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104401 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" Oct 30 06:56:20.104511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104414 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" Oct 30 06:56:20.104511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104419 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" Oct 30 06:56:20.104511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104432 5001 cpu_manager.go:414] "containerMap: removing container" podUID="822010df-d574-4cab-b041-e4f2d4fda3a2" containerName="revoke-keys" Oct 30 06:56:20.104511 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104436 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822010df-d574-4cab-b041-e4f2d4fda3a2" containerName="revoke-keys" Oct 30 06:56:20.104650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104581 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" Oct 30 06:56:20.104650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104593 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="981ae7ff-0c71-468e-b045-9f7735860591" containerName="istio-proxy" Oct 30 06:56:20.104650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104601 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="822010df-d574-4cab-b041-e4f2d4fda3a2" containerName="revoke-keys" Oct 30 06:56:20.104650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.104609 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edcda5b-e672-43ae-9fdf-10b0a644d0ae" containerName="istio-proxy" Oct 30 06:56:20.109379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.109358 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:20.129363 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.129332 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg"] Oct 30 06:56:20.134187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.134151 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:56:20.134187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.134159 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:56:20.134391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.134199 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:56:20.134391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.134208 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:56:20.276420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.276371 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-030c3ea2-vdbkg\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:20.276642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.276517 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-030c3ea2-vdbkg\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:20.276642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.276584 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjk8w\" (UniqueName: \"kubernetes.io/projected/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-kube-api-access-qjk8w\") pod \"maas-api-revoke-sub-030c3ea2-vdbkg\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:20.378068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.378029 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-030c3ea2-vdbkg\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:20.378296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.378102 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-030c3ea2-vdbkg\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:20.378296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.378140 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjk8w\" (UniqueName: \"kubernetes.io/projected/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-kube-api-access-qjk8w\") pod \"maas-api-revoke-sub-030c3ea2-vdbkg\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:20.378812 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.378791 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-030c3ea2-vdbkg\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:20.380757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.380734 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-030c3ea2-vdbkg\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:20.402135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.402111 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjk8w\" (UniqueName: \"kubernetes.io/projected/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-kube-api-access-qjk8w\") pod \"maas-api-revoke-sub-030c3ea2-vdbkg\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:20.425461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.425436 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:20.764603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:20.764563 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg"] Oct 30 06:56:21.009634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:21.009552 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" event={"ID":"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28","Type":"ContainerStarted","Data":"583cb83d8ac209403530256403ace84e442f484a59a5203613e53f588c60ceaa"} Oct 30 06:56:22.019786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:22.019750 5001 generic.go:356] "Generic (PLEG): container finished" podID="ce4cbf99-80b2-45f9-bfec-bbc0fc224a28" containerID="46460aba8717f77e69d1ad2b87aa37458d9c5a708868447c2a4f4e8f813bde8c" exitCode=0 Oct 30 06:56:22.020203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:22.019827 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" event={"ID":"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28","Type":"ContainerDied","Data":"46460aba8717f77e69d1ad2b87aa37458d9c5a708868447c2a4f4e8f813bde8c"} Oct 30 06:56:23.206212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:23.206186 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:23.221729 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:23.221696 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-openshift-ca\") pod \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " Oct 30 06:56:23.221904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:23.221744 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-kube-api-access-qjk8w\" (UniqueName: \"kubernetes.io/projected/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-kube-api-access-qjk8w\") pod \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " Oct 30 06:56:23.221904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:23.221779 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-xks-ca\") pod \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\" (UID: \"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28\") " Oct 30 06:56:23.222077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:23.222038 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-openshift-ca" pod "ce4cbf99-80b2-45f9-bfec-bbc0fc224a28" (UID: "ce4cbf99-80b2-45f9-bfec-bbc0fc224a28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:56:23.222232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:23.222218 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:23.224067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:23.224044 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-xks-ca" pod "ce4cbf99-80b2-45f9-bfec-bbc0fc224a28" (UID: "ce4cbf99-80b2-45f9-bfec-bbc0fc224a28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:56:23.224175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:23.224128 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-kube-api-access-qjk8w" pod "ce4cbf99-80b2-45f9-bfec-bbc0fc224a28" (UID: "ce4cbf99-80b2-45f9-bfec-bbc0fc224a28"). InnerVolumeSpecName "kube-api-access-qjk8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:23.323815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:23.323734 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjk8w\" (UniqueName: \"kubernetes.io/projected/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-kube-api-access-qjk8w\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:23.323815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:23.323763 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:24.038176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:24.038142 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" Oct 30 06:56:24.038404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:24.038170 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg" event={"ID":"ce4cbf99-80b2-45f9-bfec-bbc0fc224a28","Type":"ContainerDied","Data":"583cb83d8ac209403530256403ace84e442f484a59a5203613e53f588c60ceaa"} Oct 30 06:56:24.038404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:24.038264 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="583cb83d8ac209403530256403ace84e442f484a59a5203613e53f588c60ceaa" Oct 30 06:56:27.776538 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:27.776501 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:56:32.537125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.537086 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n"] Oct 30 06:56:32.537901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.537885 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ce4cbf99-80b2-45f9-bfec-bbc0fc224a28" containerName="revoke-keys" Oct 30 06:56:32.537949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.537902 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4cbf99-80b2-45f9-bfec-bbc0fc224a28" containerName="revoke-keys" Oct 30 06:56:32.538065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.538056 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4cbf99-80b2-45f9-bfec-bbc0fc224a28" containerName="revoke-keys" Oct 30 06:56:32.542277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.542257 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:32.565723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.565681 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n"] Oct 30 06:56:32.568100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.568069 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:56:32.568246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.568103 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:56:32.601105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.601066 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d180ea71-8nm2n\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:32.601332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.601125 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d180ea71-8nm2n\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:32.601332 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.601240 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4jt5\" (UniqueName: \"kubernetes.io/projected/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-kube-api-access-g4jt5\") pod \"maas-api-revoke-sub-d180ea71-8nm2n\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:32.650917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.650875 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:56:32.651072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.650942 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:56:32.702458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.702420 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d180ea71-8nm2n\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:32.702673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.702477 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d180ea71-8nm2n\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:32.702673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.702524 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4jt5\" (UniqueName: \"kubernetes.io/projected/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-kube-api-access-g4jt5\") pod \"maas-api-revoke-sub-d180ea71-8nm2n\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:32.703172 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.703147 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d180ea71-8nm2n\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:32.705058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.705034 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d180ea71-8nm2n\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:32.725551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.725527 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4jt5\" (UniqueName: \"kubernetes.io/projected/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-kube-api-access-g4jt5\") pod \"maas-api-revoke-sub-d180ea71-8nm2n\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:32.858336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:32.858288 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:33.196541 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:33.196510 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n"] Oct 30 06:56:33.197288 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:56:33.197258 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08438c69_53b5_4b5e_b614_eacbcf3d8a3d.slice/crio-6507eea50945537d1b5bda92ec547da298f26acb86c1385d4cbf71e1ec269bbb WatchSource:0}: Error finding container 6507eea50945537d1b5bda92ec547da298f26acb86c1385d4cbf71e1ec269bbb: Status 404 returned error can't find the container with id 6507eea50945537d1b5bda92ec547da298f26acb86c1385d4cbf71e1ec269bbb Oct 30 06:56:34.219483 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:34.219443 5001 generic.go:356] "Generic (PLEG): container finished" podID="08438c69-53b5-4b5e-b614-eacbcf3d8a3d" containerID="0c2dc34967013197ea6e5622b09766286bb088e276e761e437d71f4df23c928f" exitCode=0 Oct 30 06:56:34.219933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:34.219503 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" event={"ID":"08438c69-53b5-4b5e-b614-eacbcf3d8a3d","Type":"ContainerDied","Data":"0c2dc34967013197ea6e5622b09766286bb088e276e761e437d71f4df23c928f"} Oct 30 06:56:34.219933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:34.219572 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" event={"ID":"08438c69-53b5-4b5e-b614-eacbcf3d8a3d","Type":"ContainerStarted","Data":"6507eea50945537d1b5bda92ec547da298f26acb86c1385d4cbf71e1ec269bbb"} Oct 30 06:56:35.405890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:35.405857 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:35.557050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:35.556958 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-kube-api-access-g4jt5\" (UniqueName: \"kubernetes.io/projected/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-kube-api-access-g4jt5\") pod \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " Oct 30 06:56:35.557050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:35.557033 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-xks-ca\") pod \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " Oct 30 06:56:35.557385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:35.557133 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-openshift-ca\") pod \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\" (UID: \"08438c69-53b5-4b5e-b614-eacbcf3d8a3d\") " Oct 30 06:56:35.557589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:35.557562 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-openshift-ca" pod "08438c69-53b5-4b5e-b614-eacbcf3d8a3d" (UID: "08438c69-53b5-4b5e-b614-eacbcf3d8a3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:56:35.557954 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:35.557933 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:35.559518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:35.559490 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-kube-api-access-g4jt5" pod "08438c69-53b5-4b5e-b614-eacbcf3d8a3d" (UID: "08438c69-53b5-4b5e-b614-eacbcf3d8a3d"). InnerVolumeSpecName "kube-api-access-g4jt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:35.559712 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:35.559683 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-xks-ca" pod "08438c69-53b5-4b5e-b614-eacbcf3d8a3d" (UID: "08438c69-53b5-4b5e-b614-eacbcf3d8a3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:56:35.659820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:35.659786 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4jt5\" (UniqueName: \"kubernetes.io/projected/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-kube-api-access-g4jt5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:35.659820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:35.659814 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08438c69-53b5-4b5e-b614-eacbcf3d8a3d-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:36.240012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:36.239973 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" Oct 30 06:56:36.240012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:36.240003 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n" event={"ID":"08438c69-53b5-4b5e-b614-eacbcf3d8a3d","Type":"ContainerDied","Data":"6507eea50945537d1b5bda92ec547da298f26acb86c1385d4cbf71e1ec269bbb"} Oct 30 06:56:36.240246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:36.240055 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6507eea50945537d1b5bda92ec547da298f26acb86c1385d4cbf71e1ec269bbb" Oct 30 06:56:38.276291 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:56:38.276251 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:56:42.369198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:42.369159 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv"] Oct 30 06:56:42.380651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:42.380625 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c44730-hkhkv"] Oct 30 06:56:43.026860 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.026793 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs"] Oct 30 06:56:43.027760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.027735 5001 cpu_manager.go:414] "containerMap: removing container" podUID="08438c69-53b5-4b5e-b614-eacbcf3d8a3d" containerName="revoke-keys" Oct 30 06:56:43.027760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.027758 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08438c69-53b5-4b5e-b614-eacbcf3d8a3d" containerName="revoke-keys" Oct 30 06:56:43.028012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.027995 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="08438c69-53b5-4b5e-b614-eacbcf3d8a3d" containerName="revoke-keys" Oct 30 06:56:43.032158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.032135 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:43.051547 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.051514 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs"] Oct 30 06:56:43.057503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.057475 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:56:43.057503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.057476 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:56:43.102273 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.102233 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qkd\" (UniqueName: \"kubernetes.io/projected/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-kube-api-access-65qkd\") pod \"maas-api-revoke-sub-56624813-frzbs\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:43.102651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.102420 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56624813-frzbs\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:43.102651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.102448 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56624813-frzbs\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:43.103634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.103609 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:56:43.103999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.103979 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:56:43.204217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.204184 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56624813-frzbs\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:43.204447 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.204428 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qkd\" (UniqueName: \"kubernetes.io/projected/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-kube-api-access-65qkd\") pod \"maas-api-revoke-sub-56624813-frzbs\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:43.204574 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.204558 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56624813-frzbs\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:43.205202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.205175 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56624813-frzbs\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:43.206888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.206868 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56624813-frzbs\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:43.228159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.228131 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qkd\" (UniqueName: \"kubernetes.io/projected/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-kube-api-access-65qkd\") pod \"maas-api-revoke-sub-56624813-frzbs\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:43.348987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.348904 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:43.664778 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:43.664748 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs"] Oct 30 06:56:43.665904 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:56:43.665871 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b0dfc3_0c50_4813_a6c1_6f613b713c30.slice/crio-9cb6408ea923a1807ed1abe36f459d087bb5e1e981346054ea5f76077b6249c4 WatchSource:0}: Error finding container 9cb6408ea923a1807ed1abe36f459d087bb5e1e981346054ea5f76077b6249c4: Status 404 returned error can't find the container with id 9cb6408ea923a1807ed1abe36f459d087bb5e1e981346054ea5f76077b6249c4 Oct 30 06:56:44.120100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:44.120068 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822010df-d574-4cab-b041-e4f2d4fda3a2" path="/var/lib/kubelet/pods/822010df-d574-4cab-b041-e4f2d4fda3a2/volumes" Oct 30 06:56:44.326451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:44.326341 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" event={"ID":"a8b0dfc3-0c50-4813-a6c1-6f613b713c30","Type":"ContainerStarted","Data":"cf9bc1b4da74831c0819db75fc543d3f533fd7ccae2cb1b0fce9015d3d73e652"} Oct 30 06:56:44.326451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:44.326430 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" event={"ID":"a8b0dfc3-0c50-4813-a6c1-6f613b713c30","Type":"ContainerStarted","Data":"9cb6408ea923a1807ed1abe36f459d087bb5e1e981346054ea5f76077b6249c4"} Oct 30 06:56:44.371669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:44.371610 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" podStartSLOduration=1.3715837739999999 podStartE2EDuration="1.371583774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:44.365625483 +0000 UTC m=+3818.361582484" watchObservedRunningTime="2026-10-30 06:56:44.371583774 +0000 UTC m=+3818.367540789" Oct 30 06:56:45.339133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:45.339098 5001 generic.go:356] "Generic (PLEG): container finished" podID="a8b0dfc3-0c50-4813-a6c1-6f613b713c30" containerID="cf9bc1b4da74831c0819db75fc543d3f533fd7ccae2cb1b0fce9015d3d73e652" exitCode=0 Oct 30 06:56:45.339560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:45.339184 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" event={"ID":"a8b0dfc3-0c50-4813-a6c1-6f613b713c30","Type":"ContainerDied","Data":"cf9bc1b4da74831c0819db75fc543d3f533fd7ccae2cb1b0fce9015d3d73e652"} Oct 30 06:56:46.658351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:46.658300 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:46.710472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:46.710429 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-kube-api-access-65qkd\" (UniqueName: \"kubernetes.io/projected/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-kube-api-access-65qkd\") pod \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " Oct 30 06:56:46.710689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:46.710482 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-openshift-ca\") pod \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " Oct 30 06:56:46.710689 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:46.710636 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-xks-ca\") pod \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\" (UID: \"a8b0dfc3-0c50-4813-a6c1-6f613b713c30\") " Oct 30 06:56:46.711212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:46.711174 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-openshift-ca" pod "a8b0dfc3-0c50-4813-a6c1-6f613b713c30" (UID: "a8b0dfc3-0c50-4813-a6c1-6f613b713c30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:56:46.711465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:46.711438 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:46.713864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:46.713838 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-xks-ca" pod "a8b0dfc3-0c50-4813-a6c1-6f613b713c30" (UID: "a8b0dfc3-0c50-4813-a6c1-6f613b713c30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:56:46.730565 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:46.730526 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-kube-api-access-65qkd" pod "a8b0dfc3-0c50-4813-a6c1-6f613b713c30" (UID: "a8b0dfc3-0c50-4813-a6c1-6f613b713c30"). InnerVolumeSpecName "kube-api-access-65qkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:46.813134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:46.813088 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qkd\" (UniqueName: \"kubernetes.io/projected/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-kube-api-access-65qkd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:46.813134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:46.813126 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b0dfc3-0c50-4813-a6c1-6f613b713c30-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:47.039709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:47.039594 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv"] Oct 30 06:56:47.044105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:47.044059 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb49c2ad-cczkv"] Oct 30 06:56:47.365449 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:47.365417 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" Oct 30 06:56:47.365625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:47.365441 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs" event={"ID":"a8b0dfc3-0c50-4813-a6c1-6f613b713c30","Type":"ContainerDied","Data":"9cb6408ea923a1807ed1abe36f459d087bb5e1e981346054ea5f76077b6249c4"} Oct 30 06:56:47.365625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:47.365505 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cb6408ea923a1807ed1abe36f459d087bb5e1e981346054ea5f76077b6249c4" Oct 30 06:56:48.119168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:48.119136 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4038d3-ac1b-48ec-a31e-886ae343309c" path="/var/lib/kubelet/pods/db4038d3-ac1b-48ec-a31e-886ae343309c/volumes" Oct 30 06:56:49.869702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:49.869662 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm"] Oct 30 06:56:49.870209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:49.869919 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" containerID="cri-o://99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc" gracePeriod=30 Oct 30 06:56:50.488179 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.488150 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm"] Oct 30 06:56:50.488959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.488942 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a8b0dfc3-0c50-4813-a6c1-6f613b713c30" containerName="revoke-keys" Oct 30 06:56:50.489021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.488961 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b0dfc3-0c50-4813-a6c1-6f613b713c30" containerName="revoke-keys" Oct 30 06:56:50.489140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.489130 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b0dfc3-0c50-4813-a6c1-6f613b713c30" containerName="revoke-keys" Oct 30 06:56:50.494176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.494154 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:50.511648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.510132 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:56:50.511648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.510277 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:56:50.516030 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.515994 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm"] Oct 30 06:56:50.578178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.578141 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:50.578436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.578208 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:50.578436 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.578383 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjmw\" (UniqueName: \"kubernetes.io/projected/1233a018-b5df-4837-9f82-4ce065e3b0d3-kube-api-access-drjmw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:50.578688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.578671 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:56:50.578759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.578687 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:56:50.679373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.679335 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjmw\" (UniqueName: \"kubernetes.io/projected/1233a018-b5df-4837-9f82-4ce065e3b0d3-kube-api-access-drjmw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:50.679652 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.679631 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:50.679735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.679704 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:50.680408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.680388 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:50.682359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.682334 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:50.702698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.702670 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjmw\" (UniqueName: \"kubernetes.io/projected/1233a018-b5df-4837-9f82-4ce065e3b0d3-kube-api-access-drjmw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:50.814140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:50.814059 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:51.114581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.114544 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr"] Oct 30 06:56:51.120595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.120572 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:51.138686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.138664 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr"] Oct 30 06:56:51.177242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.164213 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm"] Oct 30 06:56:51.201988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.201540 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:51.201988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.201603 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlgc\" (UniqueName: \"kubernetes.io/projected/f7bb279f-bd41-468b-b0b9-30f115e82b31-kube-api-access-2rlgc\") pod \"maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:51.201988 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.201874 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:51.302888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.302852 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:51.303081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.302909 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlgc\" (UniqueName: \"kubernetes.io/projected/f7bb279f-bd41-468b-b0b9-30f115e82b31-kube-api-access-2rlgc\") pod \"maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:51.303081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.303042 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:51.303587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.303568 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:51.352616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.352578 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:51.352824 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.352744 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlgc\" (UniqueName: \"kubernetes.io/projected/f7bb279f-bd41-468b-b0b9-30f115e82b31-kube-api-access-2rlgc\") pod \"maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:51.406809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.406715 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" event={"ID":"1233a018-b5df-4837-9f82-4ce065e3b0d3","Type":"ContainerStarted","Data":"949e65c5f02ecdcdec70c1f9eae386cae055fdd9d0696724f45139d3b44bdc1c"} Oct 30 06:56:51.437738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.437707 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:51.780272 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:56:51.780239 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bb279f_bd41_468b_b0b9_30f115e82b31.slice/crio-bb532eeed719ce0de23c869badfaf611d2e4dad8e99e9df9e5b9ef7f91527ae7 WatchSource:0}: Error finding container bb532eeed719ce0de23c869badfaf611d2e4dad8e99e9df9e5b9ef7f91527ae7: Status 404 returned error can't find the container with id bb532eeed719ce0de23c869badfaf611d2e4dad8e99e9df9e5b9ef7f91527ae7 Oct 30 06:56:51.781221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:51.781194 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr"] Oct 30 06:56:52.417723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:52.417630 5001 generic.go:356] "Generic (PLEG): container finished" podID="1233a018-b5df-4837-9f82-4ce065e3b0d3" containerID="475d5363ce1891edb731f0c6111408d89faac65dc5e913fb17c64141ae7d6478" exitCode=0 Oct 30 06:56:52.417723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:52.417710 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" event={"ID":"1233a018-b5df-4837-9f82-4ce065e3b0d3","Type":"ContainerDied","Data":"475d5363ce1891edb731f0c6111408d89faac65dc5e913fb17c64141ae7d6478"} Oct 30 06:56:52.419540 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:52.419514 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" event={"ID":"f7bb279f-bd41-468b-b0b9-30f115e82b31","Type":"ContainerStarted","Data":"9467d99700a2eb8046d40f7bd939fd7a0c9ee31a9932736777d45ef502f283f7"} Oct 30 06:56:52.419663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:52.419558 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" event={"ID":"f7bb279f-bd41-468b-b0b9-30f115e82b31","Type":"ContainerStarted","Data":"bb532eeed719ce0de23c869badfaf611d2e4dad8e99e9df9e5b9ef7f91527ae7"} Oct 30 06:56:52.497327 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:52.497255 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" podStartSLOduration=1.497238183 podStartE2EDuration="1.497238183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:52.495197476 +0000 UTC m=+3826.491154488" watchObservedRunningTime="2026-10-30 06:56:52.497238183 +0000 UTC m=+3826.493195194" Oct 30 06:56:52.927268 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:56:52.927230 5001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bb279f_bd41_468b_b0b9_30f115e82b31.slice/crio-conmon-9467d99700a2eb8046d40f7bd939fd7a0c9ee31a9932736777d45ef502f283f7.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:56:53.432380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.432348 5001 generic.go:356] "Generic (PLEG): container finished" podID="f7bb279f-bd41-468b-b0b9-30f115e82b31" containerID="9467d99700a2eb8046d40f7bd939fd7a0c9ee31a9932736777d45ef502f283f7" exitCode=0 Oct 30 06:56:53.432875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.432420 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" event={"ID":"f7bb279f-bd41-468b-b0b9-30f115e82b31","Type":"ContainerDied","Data":"9467d99700a2eb8046d40f7bd939fd7a0c9ee31a9932736777d45ef502f283f7"} Oct 30 06:56:53.672735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.672707 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:53.742095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.742007 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx"] Oct 30 06:56:53.743834 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.743811 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1233a018-b5df-4837-9f82-4ce065e3b0d3" containerName="revoke-keys" Oct 30 06:56:53.743834 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.743834 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1233a018-b5df-4837-9f82-4ce065e3b0d3" containerName="revoke-keys" Oct 30 06:56:53.744193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.744151 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1233a018-b5df-4837-9f82-4ce065e3b0d3" containerName="revoke-keys" Oct 30 06:56:53.747980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.747955 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-xks-ca\") pod \"1233a018-b5df-4837-9f82-4ce065e3b0d3\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " Oct 30 06:56:53.748203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.748173 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1233a018-b5df-4837-9f82-4ce065e3b0d3-kube-api-access-drjmw\" (UniqueName: \"kubernetes.io/projected/1233a018-b5df-4837-9f82-4ce065e3b0d3-kube-api-access-drjmw\") pod \"1233a018-b5df-4837-9f82-4ce065e3b0d3\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " Oct 30 06:56:53.749261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.749173 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-openshift-ca\") pod \"1233a018-b5df-4837-9f82-4ce065e3b0d3\" (UID: \"1233a018-b5df-4837-9f82-4ce065e3b0d3\") " Oct 30 06:56:53.752153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.750658 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-openshift-ca" pod "1233a018-b5df-4837-9f82-4ce065e3b0d3" (UID: "1233a018-b5df-4837-9f82-4ce065e3b0d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:56:53.752153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.750944 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:53.770329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.757596 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx"] Oct 30 06:56:53.770329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.759338 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-xks-ca" pod "1233a018-b5df-4837-9f82-4ce065e3b0d3" (UID: "1233a018-b5df-4837-9f82-4ce065e3b0d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:56:53.770329 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.760762 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1233a018-b5df-4837-9f82-4ce065e3b0d3-kube-api-access-drjmw" pod "1233a018-b5df-4837-9f82-4ce065e3b0d3" (UID: "1233a018-b5df-4837-9f82-4ce065e3b0d3"). InnerVolumeSpecName "kube-api-access-drjmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:53.850748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.850715 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br8tj\" (UniqueName: \"kubernetes.io/projected/cc748569-b462-43b0-b82b-b1adea0b0964-kube-api-access-br8tj\") pod \"maas-api-revoke-sub-ffa253f3-wshcx\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:53.850917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.850773 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffa253f3-wshcx\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:53.851056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.851033 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffa253f3-wshcx\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:53.851157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.851146 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:53.851209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.851160 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drjmw\" (UniqueName: \"kubernetes.io/projected/1233a018-b5df-4837-9f82-4ce065e3b0d3-kube-api-access-drjmw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:53.851209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.851171 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1233a018-b5df-4837-9f82-4ce065e3b0d3-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:53.952917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.952872 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br8tj\" (UniqueName: \"kubernetes.io/projected/cc748569-b462-43b0-b82b-b1adea0b0964-kube-api-access-br8tj\") pod \"maas-api-revoke-sub-ffa253f3-wshcx\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:53.953101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.952967 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffa253f3-wshcx\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:53.953276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.953260 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffa253f3-wshcx\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:53.953869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.953839 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffa253f3-wshcx\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:53.955808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.955783 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffa253f3-wshcx\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:53.976369 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:53.976338 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br8tj\" (UniqueName: \"kubernetes.io/projected/cc748569-b462-43b0-b82b-b1adea0b0964-kube-api-access-br8tj\") pod \"maas-api-revoke-sub-ffa253f3-wshcx\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:54.109803 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.109761 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:54.426943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.426911 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx"] Oct 30 06:56:54.427535 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:56:54.427507 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc748569_b462_43b0_b82b_b1adea0b0964.slice/crio-3252f5d92b666c23c072e8ac7dc2c7276193cc7ce737c5828a8063dc66fa5c3a WatchSource:0}: Error finding container 3252f5d92b666c23c072e8ac7dc2c7276193cc7ce737c5828a8063dc66fa5c3a: Status 404 returned error can't find the container with id 3252f5d92b666c23c072e8ac7dc2c7276193cc7ce737c5828a8063dc66fa5c3a Oct 30 06:56:54.445708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.445678 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" event={"ID":"cc748569-b462-43b0-b82b-b1adea0b0964","Type":"ContainerStarted","Data":"3252f5d92b666c23c072e8ac7dc2c7276193cc7ce737c5828a8063dc66fa5c3a"} Oct 30 06:56:54.447355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.447299 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" Oct 30 06:56:54.447496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.447354 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm" event={"ID":"1233a018-b5df-4837-9f82-4ce065e3b0d3","Type":"ContainerDied","Data":"949e65c5f02ecdcdec70c1f9eae386cae055fdd9d0696724f45139d3b44bdc1c"} Oct 30 06:56:54.447496 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.447399 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="949e65c5f02ecdcdec70c1f9eae386cae055fdd9d0696724f45139d3b44bdc1c" Oct 30 06:56:54.681180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.681097 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:54.766422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.766385 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bb279f-bd41-468b-b0b9-30f115e82b31-kube-api-access-2rlgc\" (UniqueName: \"kubernetes.io/projected/f7bb279f-bd41-468b-b0b9-30f115e82b31-kube-api-access-2rlgc\") pod \"f7bb279f-bd41-468b-b0b9-30f115e82b31\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " Oct 30 06:56:54.766630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.766465 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-openshift-ca\") pod \"f7bb279f-bd41-468b-b0b9-30f115e82b31\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " Oct 30 06:56:54.766695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.766648 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-xks-ca\") pod \"f7bb279f-bd41-468b-b0b9-30f115e82b31\" (UID: \"f7bb279f-bd41-468b-b0b9-30f115e82b31\") " Oct 30 06:56:54.767107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.767076 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-openshift-ca" pod "f7bb279f-bd41-468b-b0b9-30f115e82b31" (UID: "f7bb279f-bd41-468b-b0b9-30f115e82b31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:56:54.767413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.767391 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:54.769406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.769380 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bb279f-bd41-468b-b0b9-30f115e82b31-kube-api-access-2rlgc" pod "f7bb279f-bd41-468b-b0b9-30f115e82b31" (UID: "f7bb279f-bd41-468b-b0b9-30f115e82b31"). InnerVolumeSpecName "kube-api-access-2rlgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:54.769759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.769744 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-xks-ca" pod "f7bb279f-bd41-468b-b0b9-30f115e82b31" (UID: "f7bb279f-bd41-468b-b0b9-30f115e82b31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:56:54.868859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.868832 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rlgc\" (UniqueName: \"kubernetes.io/projected/f7bb279f-bd41-468b-b0b9-30f115e82b31-kube-api-access-2rlgc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:54.868859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:54.868863 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bb279f-bd41-468b-b0b9-30f115e82b31-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:55.200208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.200177 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:56:55.285015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.284928 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-token\" (UniqueName: \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-token\") pod \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " Oct 30 06:56:55.285291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.285267 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-kube-api-access-g29km\" (UniqueName: \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-kube-api-access-g29km\") pod \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " Oct 30 06:56:55.285659 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.285636 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-podinfo\") pod \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " Oct 30 06:56:55.285958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.285936 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-certs\") pod \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " Oct 30 06:56:55.286142 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.286122 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-envoy\") pod \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " Oct 30 06:56:55.286338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.286320 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istiod-ca-cert\") pod \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " Oct 30 06:56:55.286518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.286499 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-socket\") pod \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " Oct 30 06:56:55.286699 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.286493 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-certs" pod "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" (UID: "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:56:55.286772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.286657 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-socket" pod "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" (UID: "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:56:55.286847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.286790 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istiod-ca-cert" pod "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" (UID: "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:56:55.287078 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.287018 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-data\") pod \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " Oct 30 06:56:55.287277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.287260 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-credential-socket\") pod \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\" (UID: \"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb\") " Oct 30 06:56:55.287525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.287273 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-data" pod "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" (UID: "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:56:55.287612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.287478 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-credential-socket" pod "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" (UID: "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:56:55.288351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.288330 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:55.288488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.288473 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:55.288593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.288581 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:55.288697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.288685 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:55.288797 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.288784 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:55.294174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.294147 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-envoy" pod "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" (UID: "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:56:55.295666 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.295646 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-kube-api-access-g29km" pod "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" (UID: "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb"). InnerVolumeSpecName "kube-api-access-g29km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:55.295758 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.295729 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-token" pod "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" (UID: "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:55.297253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.297233 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-podinfo" pod "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" (UID: "51a2a8f5-6f9d-4b20-ac91-eb3049e582eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:56:55.390477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.390443 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:55.390477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.390470 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g29km\" (UniqueName: \"kubernetes.io/projected/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-kube-api-access-g29km\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:55.390477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.390480 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:55.390727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.390490 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:55.536385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.533766 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" event={"ID":"f7bb279f-bd41-468b-b0b9-30f115e82b31","Type":"ContainerDied","Data":"bb532eeed719ce0de23c869badfaf611d2e4dad8e99e9df9e5b9ef7f91527ae7"} Oct 30 06:56:55.536385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.533842 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb532eeed719ce0de23c869badfaf611d2e4dad8e99e9df9e5b9ef7f91527ae7" Oct 30 06:56:55.536385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.533938 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr" Oct 30 06:56:55.545331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.539298 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" event={"ID":"cc748569-b462-43b0-b82b-b1adea0b0964","Type":"ContainerStarted","Data":"c8a6d9a5f08099abc98ab9f7430a8e5d57f733fe3b4d82896b1914ad39aeb8eb"} Oct 30 06:56:55.545331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.544719 5001 generic.go:356] "Generic (PLEG): container finished" podID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerID="99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc" exitCode=0 Oct 30 06:56:55.545331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.544763 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" event={"ID":"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb","Type":"ContainerDied","Data":"99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc"} Oct 30 06:56:55.545331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.544815 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" event={"ID":"51a2a8f5-6f9d-4b20-ac91-eb3049e582eb","Type":"ContainerDied","Data":"450337f5474b7b788ec46ef135fdcb5bf097b570bc15b4332e89227e20636e6d"} Oct 30 06:56:55.545331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.544846 5001 scope.go:122] "RemoveContainer" containerID="99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc" Oct 30 06:56:55.545331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.545037 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm" Oct 30 06:56:55.575512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.575490 5001 scope.go:122] "RemoveContainer" containerID="cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672" Oct 30 06:56:55.586375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.586277 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" podStartSLOduration=2.586258549 podStartE2EDuration="2.586258549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:55.581755037 +0000 UTC m=+3829.577712047" watchObservedRunningTime="2026-10-30 06:56:55.586258549 +0000 UTC m=+3829.582215560" Oct 30 06:56:55.592350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.592325 5001 scope.go:122] "RemoveContainer" containerID="99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc" Oct 30 06:56:55.592705 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:56:55.592679 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc\": container with ID starting with 99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc not found: ID does not exist" containerID="99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc" Oct 30 06:56:55.592790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.592719 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc"} err="failed to get container status \"99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc\": rpc error: code = NotFound desc = could not find container \"99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc\": container with ID starting with 99655b2bc50182e35e17da6b4b1326bc5b245afabd48a47f26022435b9491dfc not found: ID does not exist" Oct 30 06:56:55.592790 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.592744 5001 scope.go:122] "RemoveContainer" containerID="cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672" Oct 30 06:56:55.593042 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:56:55.593014 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672\": container with ID starting with cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672 not found: ID does not exist" containerID="cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672" Oct 30 06:56:55.593107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.593055 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672"} err="failed to get container status \"cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672\": rpc error: code = NotFound desc = could not find container \"cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672\": container with ID starting with cd56f7a37356a887812ab97f6f344de1d0f27c955a52b2adaf7102bdca348672 not found: ID does not exist" Oct 30 06:56:55.634527 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.634496 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm"] Oct 30 06:56:55.645809 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.645780 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c44730-openshift-default-7b494bd597-xd4sm"] Oct 30 06:56:55.989088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.989047 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d"] Oct 30 06:56:55.990331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.990282 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f7bb279f-bd41-468b-b0b9-30f115e82b31" containerName="revoke-keys" Oct 30 06:56:55.990331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.990323 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bb279f-bd41-468b-b0b9-30f115e82b31" containerName="revoke-keys" Oct 30 06:56:55.990460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.990347 5001 cpu_manager.go:414] "containerMap: removing container" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" Oct 30 06:56:55.990460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.990354 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" Oct 30 06:56:55.990567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.990555 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" Oct 30 06:56:55.990602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.990582 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bb279f-bd41-468b-b0b9-30f115e82b31" containerName="revoke-keys" Oct 30 06:56:55.990602 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.990595 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" Oct 30 06:56:55.995401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:55.995378 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:56.009615 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.009591 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d"] Oct 30 06:56:56.115866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.115837 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2f6e7fc-dnk5d\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:56.116025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.115989 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9zt\" (UniqueName: \"kubernetes.io/projected/07561896-3599-4430-adcf-02bff0a0b23b-kube-api-access-gh9zt\") pod \"maas-api-revoke-sub-d2f6e7fc-dnk5d\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:56.116089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.116072 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2f6e7fc-dnk5d\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:56.121174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.121148 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" path="/var/lib/kubelet/pods/51a2a8f5-6f9d-4b20-ac91-eb3049e582eb/volumes" Oct 30 06:56:56.217129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.217089 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2f6e7fc-dnk5d\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:56.217297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.217225 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9zt\" (UniqueName: \"kubernetes.io/projected/07561896-3599-4430-adcf-02bff0a0b23b-kube-api-access-gh9zt\") pod \"maas-api-revoke-sub-d2f6e7fc-dnk5d\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:56.217388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.217300 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2f6e7fc-dnk5d\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:56.217936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.217911 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2f6e7fc-dnk5d\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:56.219762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.219739 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2f6e7fc-dnk5d\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:56.240536 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.240510 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9zt\" (UniqueName: \"kubernetes.io/projected/07561896-3599-4430-adcf-02bff0a0b23b-kube-api-access-gh9zt\") pod \"maas-api-revoke-sub-d2f6e7fc-dnk5d\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:56.314826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.314780 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:56.557290 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.557252 5001 generic.go:356] "Generic (PLEG): container finished" podID="cc748569-b462-43b0-b82b-b1adea0b0964" containerID="c8a6d9a5f08099abc98ab9f7430a8e5d57f733fe3b4d82896b1914ad39aeb8eb" exitCode=0 Oct 30 06:56:56.557741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.557341 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" event={"ID":"cc748569-b462-43b0-b82b-b1adea0b0964","Type":"ContainerDied","Data":"c8a6d9a5f08099abc98ab9f7430a8e5d57f733fe3b4d82896b1914ad39aeb8eb"} Oct 30 06:56:56.701384 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:56.701267 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d"] Oct 30 06:56:56.705990 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:56:56.705960 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07561896_3599_4430_adcf_02bff0a0b23b.slice/crio-4ff08139cd1979428ad8a773d4c4025e942966f0c54c1c471f9dda40ba6da094 WatchSource:0}: Error finding container 4ff08139cd1979428ad8a773d4c4025e942966f0c54c1c471f9dda40ba6da094: Status 404 returned error can't find the container with id 4ff08139cd1979428ad8a773d4c4025e942966f0c54c1c471f9dda40ba6da094 Oct 30 06:56:57.034968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.034936 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72"] Oct 30 06:56:57.039567 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.039535 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deeba46f-tqw72"] Oct 30 06:56:57.573825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.573789 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" event={"ID":"07561896-3599-4430-adcf-02bff0a0b23b","Type":"ContainerStarted","Data":"a8c0ffd7967148c9b91ac0057c994327abe3044a93737d4b2e985f6474516d5c"} Oct 30 06:56:57.574349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.573844 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" event={"ID":"07561896-3599-4430-adcf-02bff0a0b23b","Type":"ContainerStarted","Data":"4ff08139cd1979428ad8a773d4c4025e942966f0c54c1c471f9dda40ba6da094"} Oct 30 06:56:57.632488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.632438 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" podStartSLOduration=2.632423413 podStartE2EDuration="2.632423413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:57.613621735 +0000 UTC m=+3831.609578807" watchObservedRunningTime="2026-10-30 06:56:57.632423413 +0000 UTC m=+3831.628380423" Oct 30 06:56:57.753858 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.753831 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:57.839221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.839133 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-openshift-ca\") pod \"cc748569-b462-43b0-b82b-b1adea0b0964\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " Oct 30 06:56:57.839439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.839284 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-xks-ca\") pod \"cc748569-b462-43b0-b82b-b1adea0b0964\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " Oct 30 06:56:57.839439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.839417 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc748569-b462-43b0-b82b-b1adea0b0964-kube-api-access-br8tj\" (UniqueName: \"kubernetes.io/projected/cc748569-b462-43b0-b82b-b1adea0b0964-kube-api-access-br8tj\") pod \"cc748569-b462-43b0-b82b-b1adea0b0964\" (UID: \"cc748569-b462-43b0-b82b-b1adea0b0964\") " Oct 30 06:56:57.839591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.839523 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-openshift-ca" pod "cc748569-b462-43b0-b82b-b1adea0b0964" (UID: "cc748569-b462-43b0-b82b-b1adea0b0964"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:56:57.840107 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.840083 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:57.841663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.841633 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-xks-ca" pod "cc748569-b462-43b0-b82b-b1adea0b0964" (UID: "cc748569-b462-43b0-b82b-b1adea0b0964"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:56:57.842125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.842104 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc748569-b462-43b0-b82b-b1adea0b0964-kube-api-access-br8tj" pod "cc748569-b462-43b0-b82b-b1adea0b0964" (UID: "cc748569-b462-43b0-b82b-b1adea0b0964"). InnerVolumeSpecName "kube-api-access-br8tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:57.941475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.941442 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc748569-b462-43b0-b82b-b1adea0b0964-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:57.941475 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:57.941471 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br8tj\" (UniqueName: \"kubernetes.io/projected/cc748569-b462-43b0-b82b-b1adea0b0964-kube-api-access-br8tj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:58.121043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:58.121011 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="755ec414-be8d-4a0b-93ba-a0a1229fbfbc" path="/var/lib/kubelet/pods/755ec414-be8d-4a0b-93ba-a0a1229fbfbc/volumes" Oct 30 06:56:58.584528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:58.584444 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" Oct 30 06:56:58.584528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:58.584467 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx" event={"ID":"cc748569-b462-43b0-b82b-b1adea0b0964","Type":"ContainerDied","Data":"3252f5d92b666c23c072e8ac7dc2c7276193cc7ce737c5828a8063dc66fa5c3a"} Oct 30 06:56:58.585058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:58.584528 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3252f5d92b666c23c072e8ac7dc2c7276193cc7ce737c5828a8063dc66fa5c3a" Oct 30 06:56:58.586664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:58.586613 5001 generic.go:356] "Generic (PLEG): container finished" podID="07561896-3599-4430-adcf-02bff0a0b23b" containerID="a8c0ffd7967148c9b91ac0057c994327abe3044a93737d4b2e985f6474516d5c" exitCode=0 Oct 30 06:56:58.586831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:58.586722 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" event={"ID":"07561896-3599-4430-adcf-02bff0a0b23b","Type":"ContainerDied","Data":"a8c0ffd7967148c9b91ac0057c994327abe3044a93737d4b2e985f6474516d5c"} Oct 30 06:56:59.793117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:59.793091 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:56:59.888830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:59.888796 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-xks-ca\") pod \"07561896-3599-4430-adcf-02bff0a0b23b\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " Oct 30 06:56:59.889072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:59.888877 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-openshift-ca\") pod \"07561896-3599-4430-adcf-02bff0a0b23b\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " Oct 30 06:56:59.889072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:59.888907 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07561896-3599-4430-adcf-02bff0a0b23b-kube-api-access-gh9zt\" (UniqueName: \"kubernetes.io/projected/07561896-3599-4430-adcf-02bff0a0b23b-kube-api-access-gh9zt\") pod \"07561896-3599-4430-adcf-02bff0a0b23b\" (UID: \"07561896-3599-4430-adcf-02bff0a0b23b\") " Oct 30 06:56:59.889261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:59.889228 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-openshift-ca" pod "07561896-3599-4430-adcf-02bff0a0b23b" (UID: "07561896-3599-4430-adcf-02bff0a0b23b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:56:59.889614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:59.889590 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:59.891250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:59.891227 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-xks-ca" pod "07561896-3599-4430-adcf-02bff0a0b23b" (UID: "07561896-3599-4430-adcf-02bff0a0b23b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:56:59.891401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:59.891318 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07561896-3599-4430-adcf-02bff0a0b23b-kube-api-access-gh9zt" pod "07561896-3599-4430-adcf-02bff0a0b23b" (UID: "07561896-3599-4430-adcf-02bff0a0b23b"). InnerVolumeSpecName "kube-api-access-gh9zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:59.990949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:59.990914 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh9zt\" (UniqueName: \"kubernetes.io/projected/07561896-3599-4430-adcf-02bff0a0b23b-kube-api-access-gh9zt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:56:59.990949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:56:59.990943 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07561896-3599-4430-adcf-02bff0a0b23b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:00.649846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:00.649804 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" event={"ID":"07561896-3599-4430-adcf-02bff0a0b23b","Type":"ContainerDied","Data":"4ff08139cd1979428ad8a773d4c4025e942966f0c54c1c471f9dda40ba6da094"} Oct 30 06:57:00.650026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:00.649866 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ff08139cd1979428ad8a773d4c4025e942966f0c54c1c471f9dda40ba6da094" Oct 30 06:57:00.650026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:00.649814 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d" Oct 30 06:57:00.981129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:00.981042 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd"] Oct 30 06:57:01.019320 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:01.019276 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6"] Oct 30 06:57:01.776797 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:01.776505 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4"] Oct 30 06:57:01.783617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:01.783581 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps"] Oct 30 06:57:05.453158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:05.453118 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt"] Oct 30 06:57:05.981765 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:05.981728 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" podUID="1c77805f-d7de-44d8-8a27-3c910cb60b4e" containerName="payload-processing" containerID="cri-o://ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721" gracePeriod=25 Oct 30 06:57:06.019996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.019926 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" podUID="791a33a9-9513-4bd6-bd64-0c37766184df" containerName="payload-pre-processing" containerID="cri-o://ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40" gracePeriod=25 Oct 30 06:57:06.048811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.048777 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx"] Oct 30 06:57:06.049595 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.049580 5001 cpu_manager.go:414] "containerMap: removing container" podUID="07561896-3599-4430-adcf-02bff0a0b23b" containerName="revoke-keys" Oct 30 06:57:06.049657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.049597 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07561896-3599-4430-adcf-02bff0a0b23b" containerName="revoke-keys" Oct 30 06:57:06.049657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.049613 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cc748569-b462-43b0-b82b-b1adea0b0964" containerName="revoke-keys" Oct 30 06:57:06.049657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.049618 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc748569-b462-43b0-b82b-b1adea0b0964" containerName="revoke-keys" Oct 30 06:57:06.049657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.049633 5001 cpu_manager.go:414] "containerMap: removing container" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" Oct 30 06:57:06.049657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.049638 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a2a8f5-6f9d-4b20-ac91-eb3049e582eb" containerName="istio-proxy" Oct 30 06:57:06.049828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.049782 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="07561896-3599-4430-adcf-02bff0a0b23b" containerName="revoke-keys" Oct 30 06:57:06.049828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.049790 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc748569-b462-43b0-b82b-b1adea0b0964" containerName="revoke-keys" Oct 30 06:57:06.053895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.053876 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:06.073931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.073900 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx"] Oct 30 06:57:06.079121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.079090 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:57:06.079264 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.079137 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:57:06.117717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.117694 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:57:06.117903 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.117841 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:57:06.141413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.141350 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vd4x\" (UniqueName: \"kubernetes.io/projected/3364fda6-e33a-4ada-abfa-84cba1506d16-kube-api-access-6vd4x\") pod \"maas-api-revoke-sub-c011337f-b7gkx\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:06.141638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.141472 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c011337f-b7gkx\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:06.153993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.151630 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c011337f-b7gkx\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:06.253478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.253389 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vd4x\" (UniqueName: \"kubernetes.io/projected/3364fda6-e33a-4ada-abfa-84cba1506d16-kube-api-access-6vd4x\") pod \"maas-api-revoke-sub-c011337f-b7gkx\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:06.253663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.253474 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c011337f-b7gkx\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:06.253663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.253632 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c011337f-b7gkx\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:06.266776 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.266744 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:57:06.266928 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.266827 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:57:06.274613 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.274582 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c011337f-b7gkx\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:06.276077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.276047 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c011337f-b7gkx\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:06.286836 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.286812 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:57:06.297007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.296981 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vd4x\" (UniqueName: \"kubernetes.io/projected/3364fda6-e33a-4ada-abfa-84cba1506d16-kube-api-access-6vd4x\") pod \"maas-api-revoke-sub-c011337f-b7gkx\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:06.382910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.382886 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:57:06.390220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.390197 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:06.709414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.709390 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx"] Oct 30 06:57:06.710503 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:57:06.710472 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3364fda6_e33a_4ada_abfa_84cba1506d16.slice/crio-847f63295bfd71a99a8840573756738f9b3ba2067ebe41c6d92a2dcfbb8f9161 WatchSource:0}: Error finding container 847f63295bfd71a99a8840573756738f9b3ba2067ebe41c6d92a2dcfbb8f9161: Status 404 returned error can't find the container with id 847f63295bfd71a99a8840573756738f9b3ba2067ebe41c6d92a2dcfbb8f9161 Oct 30 06:57:06.734817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.734788 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" event={"ID":"3364fda6-e33a-4ada-abfa-84cba1506d16","Type":"ContainerStarted","Data":"847f63295bfd71a99a8840573756738f9b3ba2067ebe41c6d92a2dcfbb8f9161"} Oct 30 06:57:06.777135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.777042 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" podUID="ad7d8325-1c73-43af-bea6-5f14fe875bdf" containerName="payload-processing" containerID="cri-o://5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99" gracePeriod=25 Oct 30 06:57:06.784269 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:06.784239 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" podUID="f616e0d8-f098-4e17-8dfc-ce9076813eb0" containerName="payload-pre-processing" containerID="cri-o://11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe" gracePeriod=25 Oct 30 06:57:07.059899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.059865 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:57:07.165714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.165681 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad7d8325-1c73-43af-bea6-5f14fe875bdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7d8325-1c73-43af-bea6-5f14fe875bdf-plugins-config-volume\") pod \"ad7d8325-1c73-43af-bea6-5f14fe875bdf\" (UID: \"ad7d8325-1c73-43af-bea6-5f14fe875bdf\") " Oct 30 06:57:07.165925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.165905 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7d8325-1c73-43af-bea6-5f14fe875bdf-kube-api-access-5ld7s\" (UniqueName: \"kubernetes.io/projected/ad7d8325-1c73-43af-bea6-5f14fe875bdf-kube-api-access-5ld7s\") pod \"ad7d8325-1c73-43af-bea6-5f14fe875bdf\" (UID: \"ad7d8325-1c73-43af-bea6-5f14fe875bdf\") " Oct 30 06:57:07.166123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.166102 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7d8325-1c73-43af-bea6-5f14fe875bdf-plugins-config-volume" pod "ad7d8325-1c73-43af-bea6-5f14fe875bdf" (UID: "ad7d8325-1c73-43af-bea6-5f14fe875bdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:07.166593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.166570 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7d8325-1c73-43af-bea6-5f14fe875bdf-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:07.168232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.168209 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7d8325-1c73-43af-bea6-5f14fe875bdf-kube-api-access-5ld7s" pod "ad7d8325-1c73-43af-bea6-5f14fe875bdf" (UID: "ad7d8325-1c73-43af-bea6-5f14fe875bdf"). InnerVolumeSpecName "kube-api-access-5ld7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:07.268606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.268518 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ld7s\" (UniqueName: \"kubernetes.io/projected/ad7d8325-1c73-43af-bea6-5f14fe875bdf-kube-api-access-5ld7s\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:07.745296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.745263 5001 generic.go:356] "Generic (PLEG): container finished" podID="ad7d8325-1c73-43af-bea6-5f14fe875bdf" containerID="5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99" exitCode=0 Oct 30 06:57:07.745296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.745320 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" Oct 30 06:57:07.745838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.745357 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" event={"ID":"ad7d8325-1c73-43af-bea6-5f14fe875bdf","Type":"ContainerDied","Data":"5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99"} Oct 30 06:57:07.745838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.745415 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4" event={"ID":"ad7d8325-1c73-43af-bea6-5f14fe875bdf","Type":"ContainerDied","Data":"0cf86613f988da247b3fda614d2f2140e7349706a049d743561d10cb08846ce8"} Oct 30 06:57:07.745838 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.745446 5001 scope.go:122] "RemoveContainer" containerID="5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99" Oct 30 06:57:07.747382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.747355 5001 generic.go:356] "Generic (PLEG): container finished" podID="3364fda6-e33a-4ada-abfa-84cba1506d16" containerID="1bac391679d7eba1223af922b2d5a09af50597131a26c88044860c98ddcb2de1" exitCode=0 Oct 30 06:57:07.747522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.747485 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" event={"ID":"3364fda6-e33a-4ada-abfa-84cba1506d16","Type":"ContainerDied","Data":"1bac391679d7eba1223af922b2d5a09af50597131a26c88044860c98ddcb2de1"} Oct 30 06:57:07.759510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.759489 5001 scope.go:122] "RemoveContainer" containerID="5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99" Oct 30 06:57:07.759889 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:07.759865 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99\": container with ID starting with 5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99 not found: ID does not exist" containerID="5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99" Oct 30 06:57:07.759983 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.759909 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99"} err="failed to get container status \"5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99\": rpc error: code = NotFound desc = could not find container \"5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99\": container with ID starting with 5435009a6abae25301ca92813d90538b1e097720875a6f934ae148a8a96c5f99 not found: ID does not exist" Oct 30 06:57:07.840695 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.840662 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4"] Oct 30 06:57:07.845338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:07.845296 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b22a13-59d6b5f474-sxgb4"] Oct 30 06:57:08.119945 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:08.119899 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad7d8325-1c73-43af-bea6-5f14fe875bdf" path="/var/lib/kubelet/pods/ad7d8325-1c73-43af-bea6-5f14fe875bdf/volumes" Oct 30 06:57:08.267732 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:08.267695 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:57:08.944716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:08.944685 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:09.094276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.094180 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-xks-ca\") pod \"3364fda6-e33a-4ada-abfa-84cba1506d16\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " Oct 30 06:57:09.094501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.094325 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3364fda6-e33a-4ada-abfa-84cba1506d16-kube-api-access-6vd4x\" (UniqueName: \"kubernetes.io/projected/3364fda6-e33a-4ada-abfa-84cba1506d16-kube-api-access-6vd4x\") pod \"3364fda6-e33a-4ada-abfa-84cba1506d16\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " Oct 30 06:57:09.094501 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.094399 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-openshift-ca\") pod \"3364fda6-e33a-4ada-abfa-84cba1506d16\" (UID: \"3364fda6-e33a-4ada-abfa-84cba1506d16\") " Oct 30 06:57:09.094804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.094779 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-openshift-ca" pod "3364fda6-e33a-4ada-abfa-84cba1506d16" (UID: "3364fda6-e33a-4ada-abfa-84cba1506d16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:09.094997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.094971 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:09.096556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.096531 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-xks-ca" pod "3364fda6-e33a-4ada-abfa-84cba1506d16" (UID: "3364fda6-e33a-4ada-abfa-84cba1506d16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:09.096691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.096676 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3364fda6-e33a-4ada-abfa-84cba1506d16-kube-api-access-6vd4x" pod "3364fda6-e33a-4ada-abfa-84cba1506d16" (UID: "3364fda6-e33a-4ada-abfa-84cba1506d16"). InnerVolumeSpecName "kube-api-access-6vd4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:09.196955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.196919 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3364fda6-e33a-4ada-abfa-84cba1506d16-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:09.196955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.196951 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vd4x\" (UniqueName: \"kubernetes.io/projected/3364fda6-e33a-4ada-abfa-84cba1506d16-kube-api-access-6vd4x\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:09.786360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.785379 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" event={"ID":"3364fda6-e33a-4ada-abfa-84cba1506d16","Type":"ContainerDied","Data":"847f63295bfd71a99a8840573756738f9b3ba2067ebe41c6d92a2dcfbb8f9161"} Oct 30 06:57:09.786360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.785555 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="847f63295bfd71a99a8840573756738f9b3ba2067ebe41c6d92a2dcfbb8f9161" Oct 30 06:57:09.786360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:09.785599 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx" Oct 30 06:57:11.926193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:11.926157 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm"] Oct 30 06:57:11.938187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:11.938023 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4f3973-g8csm"] Oct 30 06:57:11.969330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:11.966940 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" podUID="791a33a9-9513-4bd6-bd64-0c37766184df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:12.119518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:12.119488 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1233a018-b5df-4837-9f82-4ce065e3b0d3" path="/var/lib/kubelet/pods/1233a018-b5df-4837-9f82-4ce065e3b0d3/volumes" Oct 30 06:57:12.465322 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:12.465265 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" podUID="f616e0d8-f098-4e17-8dfc-ce9076813eb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:13.147330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:13.139957 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" podUID="1c77805f-d7de-44d8-8a27-3c910cb60b4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:14.282794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:14.282758 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr"] Oct 30 06:57:14.291338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:14.291291 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b22a13-25jlr"] Oct 30 06:57:16.121166 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:16.121133 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bb279f-bd41-468b-b0b9-30f115e82b31" path="/var/lib/kubelet/pods/f7bb279f-bd41-468b-b0b9-30f115e82b31/volumes" Oct 30 06:57:18.499377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:18.499343 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:57:19.727038 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.724937 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw"] Oct 30 06:57:19.729324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.729251 5001 cpu_manager.go:414] "containerMap: removing container" podUID="3364fda6-e33a-4ada-abfa-84cba1506d16" containerName="revoke-keys" Oct 30 06:57:19.729522 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.729503 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3364fda6-e33a-4ada-abfa-84cba1506d16" containerName="revoke-keys" Oct 30 06:57:19.729686 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.729651 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ad7d8325-1c73-43af-bea6-5f14fe875bdf" containerName="payload-processing" Oct 30 06:57:19.729794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.729781 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7d8325-1c73-43af-bea6-5f14fe875bdf" containerName="payload-processing" Oct 30 06:57:19.730178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.730161 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3364fda6-e33a-4ada-abfa-84cba1506d16" containerName="revoke-keys" Oct 30 06:57:19.730298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.730287 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7d8325-1c73-43af-bea6-5f14fe875bdf" containerName="payload-processing" Oct 30 06:57:19.736965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.736944 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:19.775364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.763243 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:57:19.775665 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.763327 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:57:19.803334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.789722 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:57:19.803334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.789804 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:57:19.803334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.794111 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw"] Oct 30 06:57:19.852379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.852340 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6l9s\" (UniqueName: \"kubernetes.io/projected/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-kube-api-access-l6l9s\") pod \"maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:19.852723 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.852701 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:19.880331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.859807 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:20.005382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.997026 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:20.005382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.997129 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6l9s\" (UniqueName: \"kubernetes.io/projected/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-kube-api-access-l6l9s\") pod \"maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:20.005382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:19.997261 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:20.005382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.004811 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:20.077759 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.077688 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:20.130941 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.122597 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6l9s\" (UniqueName: \"kubernetes.io/projected/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-kube-api-access-l6l9s\") pod \"maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:20.149679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.149320 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:20.524122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.524059 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" podUID="6483839b-0d24-4763-b6c8-10337a352e0a" containerName="main" containerID="cri-o://4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8" gracePeriod=45 Oct 30 06:57:20.661004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.660973 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm"] Oct 30 06:57:20.667650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.667623 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:20.685718 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.685679 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm"] Oct 30 06:57:20.708640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.708586 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:20.708854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.708729 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:20.708998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.708979 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b27fs\" (UniqueName: \"kubernetes.io/projected/f9f0337b-5822-4a2f-83fc-9bc5c229124b-kube-api-access-b27fs\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:20.810479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.810393 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:20.810479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.810468 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:20.811041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.810666 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b27fs\" (UniqueName: \"kubernetes.io/projected/f9f0337b-5822-4a2f-83fc-9bc5c229124b-kube-api-access-b27fs\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:20.811255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.811227 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:20.813341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.813239 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:20.826951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.826823 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw"] Oct 30 06:57:20.827681 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:57:20.827651 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0e0d65_dc76_4fd9_ad98_9a77158f09f8.slice/crio-5cd8bcebbd6a9afc0b80db1a3f541ea12dca840b915283d7feff1dc40dfe5831 WatchSource:0}: Error finding container 5cd8bcebbd6a9afc0b80db1a3f541ea12dca840b915283d7feff1dc40dfe5831: Status 404 returned error can't find the container with id 5cd8bcebbd6a9afc0b80db1a3f541ea12dca840b915283d7feff1dc40dfe5831 Oct 30 06:57:20.834209 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.834187 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b27fs\" (UniqueName: \"kubernetes.io/projected/f9f0337b-5822-4a2f-83fc-9bc5c229124b-kube-api-access-b27fs\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:20.844757 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.844729 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:57:20.911385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.911344 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-tmp-dir\") pod \"6483839b-0d24-4763-b6c8-10337a352e0a\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " Oct 30 06:57:20.911576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.911440 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-dshm\") pod \"6483839b-0d24-4763-b6c8-10337a352e0a\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " Oct 30 06:57:20.911662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.911586 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6483839b-0d24-4763-b6c8-10337a352e0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/6483839b-0d24-4763-b6c8-10337a352e0a-tls-certs\") pod \"6483839b-0d24-4763-b6c8-10337a352e0a\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " Oct 30 06:57:20.911742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.911578 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-tmp-dir" pod "6483839b-0d24-4763-b6c8-10337a352e0a" (UID: "6483839b-0d24-4763-b6c8-10337a352e0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:20.911808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.911724 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-home\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-home\") pod \"6483839b-0d24-4763-b6c8-10337a352e0a\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " Oct 30 06:57:20.911866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.911809 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6483839b-0d24-4763-b6c8-10337a352e0a-kube-api-access-7hgpf\" (UniqueName: \"kubernetes.io/projected/6483839b-0d24-4763-b6c8-10337a352e0a-kube-api-access-7hgpf\") pod \"6483839b-0d24-4763-b6c8-10337a352e0a\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " Oct 30 06:57:20.911866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.911843 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-model-cache\") pod \"6483839b-0d24-4763-b6c8-10337a352e0a\" (UID: \"6483839b-0d24-4763-b6c8-10337a352e0a\") " Oct 30 06:57:20.911997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.911951 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-home" pod "6483839b-0d24-4763-b6c8-10337a352e0a" (UID: "6483839b-0d24-4763-b6c8-10337a352e0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:20.912224 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.912191 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-model-cache" pod "6483839b-0d24-4763-b6c8-10337a352e0a" (UID: "6483839b-0d24-4763-b6c8-10337a352e0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:20.912598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.912574 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:20.912687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.912604 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:20.912687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.912619 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:20.914277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.914241 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6483839b-0d24-4763-b6c8-10337a352e0a-tls-certs" pod "6483839b-0d24-4763-b6c8-10337a352e0a" (UID: "6483839b-0d24-4763-b6c8-10337a352e0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:20.914395 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.914293 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6483839b-0d24-4763-b6c8-10337a352e0a-kube-api-access-7hgpf" pod "6483839b-0d24-4763-b6c8-10337a352e0a" (UID: "6483839b-0d24-4763-b6c8-10337a352e0a"). InnerVolumeSpecName "kube-api-access-7hgpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:20.914800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.914778 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-dshm" pod "6483839b-0d24-4763-b6c8-10337a352e0a" (UID: "6483839b-0d24-4763-b6c8-10337a352e0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:20.965794 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.965762 5001 generic.go:356] "Generic (PLEG): container finished" podID="6483839b-0d24-4763-b6c8-10337a352e0a" containerID="4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8" exitCode=0 Oct 30 06:57:20.965970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.965811 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" Oct 30 06:57:20.965970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.965851 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" event={"ID":"6483839b-0d24-4763-b6c8-10337a352e0a","Type":"ContainerDied","Data":"4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8"} Oct 30 06:57:20.965970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.965903 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt" event={"ID":"6483839b-0d24-4763-b6c8-10337a352e0a","Type":"ContainerDied","Data":"23f0cdad2da2b9e1625b1bd59bd35d8c3aa58bc7d7eabf73770775910e92d76e"} Oct 30 06:57:20.965970 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.965922 5001 scope.go:122] "RemoveContainer" containerID="4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8" Oct 30 06:57:20.967767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.967743 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" event={"ID":"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8","Type":"ContainerStarted","Data":"5cd8bcebbd6a9afc0b80db1a3f541ea12dca840b915283d7feff1dc40dfe5831"} Oct 30 06:57:20.980080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.980058 5001 scope.go:122] "RemoveContainer" containerID="4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8" Oct 30 06:57:20.980460 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:20.980435 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8\": container with ID starting with 4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8 not found: ID does not exist" containerID="4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8" Oct 30 06:57:20.980572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.980469 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8"} err="failed to get container status \"4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8\": rpc error: code = NotFound desc = could not find container \"4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8\": container with ID starting with 4100f4220a839efd87c4f6d229ff5b7f3d42e0942a3ac505b034878af347ddf8 not found: ID does not exist" Oct 30 06:57:20.999184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:20.999151 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:21.014682 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:21.014647 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hgpf\" (UniqueName: \"kubernetes.io/projected/6483839b-0d24-4763-b6c8-10337a352e0a-kube-api-access-7hgpf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:21.014914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:21.014685 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6483839b-0d24-4763-b6c8-10337a352e0a-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:21.014914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:21.014885 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6483839b-0d24-4763-b6c8-10337a352e0a-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:21.021507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:21.021482 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt"] Oct 30 06:57:21.029439 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:21.029414 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8baf7e/test-model-8baf7e-kserve-676bf47589-pjmwt"] Oct 30 06:57:21.331581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:21.331551 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm"] Oct 30 06:57:21.333896 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:57:21.333869 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f0337b_5822_4a2f_83fc_9bc5c229124b.slice/crio-24c1a7ced76b095704a84bf8ed53acf2f7e82cc6d69549f003b586c45b9a54bb WatchSource:0}: Error finding container 24c1a7ced76b095704a84bf8ed53acf2f7e82cc6d69549f003b586c45b9a54bb: Status 404 returned error can't find the container with id 24c1a7ced76b095704a84bf8ed53acf2f7e82cc6d69549f003b586c45b9a54bb Oct 30 06:57:21.775400 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:21.774123 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs"] Oct 30 06:57:22.016429 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.006592 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" podUID="791a33a9-9513-4bd6-bd64-0c37766184df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:22.026219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.023780 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" event={"ID":"f9f0337b-5822-4a2f-83fc-9bc5c229124b","Type":"ContainerStarted","Data":"a7e7715f9be3436997a4768111511885dd3b32eb0764db8f3ced9538235fd043"} Oct 30 06:57:22.026219 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.023846 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" event={"ID":"f9f0337b-5822-4a2f-83fc-9bc5c229124b","Type":"ContainerStarted","Data":"24c1a7ced76b095704a84bf8ed53acf2f7e82cc6d69549f003b586c45b9a54bb"} Oct 30 06:57:22.026982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.026948 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" event={"ID":"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8","Type":"ContainerStarted","Data":"e37e27d078a933b9c4cea6551298adc175953f068f17e3982a89b3ee7274a5d8"} Oct 30 06:57:22.056435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.056396 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8"] Oct 30 06:57:22.057457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.057431 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6483839b-0d24-4763-b6c8-10337a352e0a" containerName="main" Oct 30 06:57:22.057457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.057457 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6483839b-0d24-4763-b6c8-10337a352e0a" containerName="main" Oct 30 06:57:22.057656 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.057638 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6483839b-0d24-4763-b6c8-10337a352e0a" containerName="main" Oct 30 06:57:22.061652 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.061633 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:22.063914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.063867 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" podStartSLOduration=2.063855746 podStartE2EDuration="2.063855746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:22.062954618 +0000 UTC m=+3856.058911628" watchObservedRunningTime="2026-10-30 06:57:22.063855746 +0000 UTC m=+3856.059812756" Oct 30 06:57:22.082739 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.082712 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8"] Oct 30 06:57:22.133092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.133034 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:22.136558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.133463 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:22.136558 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.133571 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6qw7\" (UniqueName: \"kubernetes.io/projected/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-kube-api-access-c6qw7\") pod \"maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:22.138593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.138558 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6483839b-0d24-4763-b6c8-10337a352e0a" path="/var/lib/kubelet/pods/6483839b-0d24-4763-b6c8-10337a352e0a/volumes" Oct 30 06:57:22.146822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.146776 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" podStartSLOduration=3.146764444 podStartE2EDuration="3.146764444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:22.142367717 +0000 UTC m=+3856.138324727" watchObservedRunningTime="2026-10-30 06:57:22.146764444 +0000 UTC m=+3856.142721453" Oct 30 06:57:22.238889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.238846 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:22.239088 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.238958 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6qw7\" (UniqueName: \"kubernetes.io/projected/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-kube-api-access-c6qw7\") pod \"maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:22.239144 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.239127 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:22.239592 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.239570 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:22.241706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.241684 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:22.262190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.262161 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6qw7\" (UniqueName: \"kubernetes.io/projected/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-kube-api-access-c6qw7\") pod \"maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:22.444875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.444843 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:22.465982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.465947 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" podUID="f616e0d8-f098-4e17-8dfc-ce9076813eb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:22.758243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:22.758213 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8"] Oct 30 06:57:22.759554 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:57:22.759522 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b9c3f0_9eda_481c_bf6a_86f72cb837fd.slice/crio-581d055e6f9d0e64a72bdb3f28f6009ed2f1871c25e5e33542fca49041e83353 WatchSource:0}: Error finding container 581d055e6f9d0e64a72bdb3f28f6009ed2f1871c25e5e33542fca49041e83353: Status 404 returned error can't find the container with id 581d055e6f9d0e64a72bdb3f28f6009ed2f1871c25e5e33542fca49041e83353 Oct 30 06:57:23.037772 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:23.037680 5001 generic.go:356] "Generic (PLEG): container finished" podID="f9f0337b-5822-4a2f-83fc-9bc5c229124b" containerID="a7e7715f9be3436997a4768111511885dd3b32eb0764db8f3ced9538235fd043" exitCode=0 Oct 30 06:57:23.038199 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:23.037764 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" event={"ID":"f9f0337b-5822-4a2f-83fc-9bc5c229124b","Type":"ContainerDied","Data":"a7e7715f9be3436997a4768111511885dd3b32eb0764db8f3ced9538235fd043"} Oct 30 06:57:23.039675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:23.039643 5001 generic.go:356] "Generic (PLEG): container finished" podID="ae0e0d65-dc76-4fd9-ad98-9a77158f09f8" containerID="e37e27d078a933b9c4cea6551298adc175953f068f17e3982a89b3ee7274a5d8" exitCode=0 Oct 30 06:57:23.039817 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:23.039727 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" event={"ID":"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8","Type":"ContainerDied","Data":"e37e27d078a933b9c4cea6551298adc175953f068f17e3982a89b3ee7274a5d8"} Oct 30 06:57:23.041198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:23.041179 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" event={"ID":"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd","Type":"ContainerStarted","Data":"581d055e6f9d0e64a72bdb3f28f6009ed2f1871c25e5e33542fca49041e83353"} Oct 30 06:57:23.140634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:23.140591 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" podUID="1c77805f-d7de-44d8-8a27-3c910cb60b4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:23.398544 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:23.398508 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv"] Oct 30 06:57:23.398813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:23.398773 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" containerID="cri-o://33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77" gracePeriod=30 Oct 30 06:57:24.031428 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.031345 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr"] Oct 30 06:57:24.036218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.036199 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:24.053960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.053929 5001 generic.go:356] "Generic (PLEG): container finished" podID="d9b9c3f0-9eda-481c-bf6a-86f72cb837fd" containerID="d1901bd0523d9f2abfdf5f2764bce1aea5923187aee3ba0a0f06af2eaef1ded4" exitCode=0 Oct 30 06:57:24.054469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.054009 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" event={"ID":"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd","Type":"ContainerDied","Data":"d1901bd0523d9f2abfdf5f2764bce1aea5923187aee3ba0a0f06af2eaef1ded4"} Oct 30 06:57:24.056404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.056369 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr"] Oct 30 06:57:24.201667 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.201626 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:24.201845 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.201720 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcdw7\" (UniqueName: \"kubernetes.io/projected/9702025a-0ba2-4515-9118-d700256f7aa8-kube-api-access-lcdw7\") pod \"maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:24.201956 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.201935 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:24.271083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.271053 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:24.303777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.303748 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcdw7\" (UniqueName: \"kubernetes.io/projected/9702025a-0ba2-4515-9118-d700256f7aa8-kube-api-access-lcdw7\") pod \"maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:24.303949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.303902 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:24.304051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.304029 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:24.304823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.304754 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:24.306539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.306518 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:24.327212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.327181 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcdw7\" (UniqueName: \"kubernetes.io/projected/9702025a-0ba2-4515-9118-d700256f7aa8-kube-api-access-lcdw7\") pod \"maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:24.335792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.335770 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:24.352164 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.352137 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:24.404828 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.404792 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f0337b-5822-4a2f-83fc-9bc5c229124b-kube-api-access-b27fs\" (UniqueName: \"kubernetes.io/projected/f9f0337b-5822-4a2f-83fc-9bc5c229124b-kube-api-access-b27fs\") pod \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " Oct 30 06:57:24.405017 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.404929 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-kube-api-access-l6l9s\" (UniqueName: \"kubernetes.io/projected/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-kube-api-access-l6l9s\") pod \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " Oct 30 06:57:24.405156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.405016 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-openshift-ca\") pod \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " Oct 30 06:57:24.405331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.405199 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-xks-ca\") pod \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " Oct 30 06:57:24.405331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.405287 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-openshift-ca\") pod \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\" (UID: \"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8\") " Oct 30 06:57:24.405478 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.405390 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-xks-ca\") pod \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\" (UID: \"f9f0337b-5822-4a2f-83fc-9bc5c229124b\") " Oct 30 06:57:24.406348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.405897 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-openshift-ca" pod "f9f0337b-5822-4a2f-83fc-9bc5c229124b" (UID: "f9f0337b-5822-4a2f-83fc-9bc5c229124b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:24.406348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.406132 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:24.406535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.406471 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-openshift-ca" pod "ae0e0d65-dc76-4fd9-ad98-9a77158f09f8" (UID: "ae0e0d65-dc76-4fd9-ad98-9a77158f09f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:24.407957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.407928 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f0337b-5822-4a2f-83fc-9bc5c229124b-kube-api-access-b27fs" pod "f9f0337b-5822-4a2f-83fc-9bc5c229124b" (UID: "f9f0337b-5822-4a2f-83fc-9bc5c229124b"). InnerVolumeSpecName "kube-api-access-b27fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:24.408693 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.408668 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-kube-api-access-l6l9s" pod "ae0e0d65-dc76-4fd9-ad98-9a77158f09f8" (UID: "ae0e0d65-dc76-4fd9-ad98-9a77158f09f8"). InnerVolumeSpecName "kube-api-access-l6l9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:24.409194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.409166 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-xks-ca" pod "ae0e0d65-dc76-4fd9-ad98-9a77158f09f8" (UID: "ae0e0d65-dc76-4fd9-ad98-9a77158f09f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:24.411260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.410676 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-xks-ca" pod "f9f0337b-5822-4a2f-83fc-9bc5c229124b" (UID: "f9f0337b-5822-4a2f-83fc-9bc5c229124b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:24.507488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.507455 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:24.507488 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.507489 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:24.507704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.507500 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f0337b-5822-4a2f-83fc-9bc5c229124b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:24.507704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.507509 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b27fs\" (UniqueName: \"kubernetes.io/projected/f9f0337b-5822-4a2f-83fc-9bc5c229124b-kube-api-access-b27fs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:24.507704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.507521 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6l9s\" (UniqueName: \"kubernetes.io/projected/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8-kube-api-access-l6l9s\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:24.679605 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:57:24.679566 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9702025a_0ba2_4515_9118_d700256f7aa8.slice/crio-49e62e236806b425ace276afd8a64f3c72e19ecb06d06581407b8ff10dbeb9b0 WatchSource:0}: Error finding container 49e62e236806b425ace276afd8a64f3c72e19ecb06d06581407b8ff10dbeb9b0: Status 404 returned error can't find the container with id 49e62e236806b425ace276afd8a64f3c72e19ecb06d06581407b8ff10dbeb9b0 Oct 30 06:57:24.680608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:24.680579 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr"] Oct 30 06:57:25.066756 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.066708 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" event={"ID":"f9f0337b-5822-4a2f-83fc-9bc5c229124b","Type":"ContainerDied","Data":"24c1a7ced76b095704a84bf8ed53acf2f7e82cc6d69549f003b586c45b9a54bb"} Oct 30 06:57:25.067243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.066775 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24c1a7ced76b095704a84bf8ed53acf2f7e82cc6d69549f003b586c45b9a54bb" Oct 30 06:57:25.067243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.066774 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm" Oct 30 06:57:25.068591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.068568 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" Oct 30 06:57:25.068741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.068626 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw" event={"ID":"ae0e0d65-dc76-4fd9-ad98-9a77158f09f8","Type":"ContainerDied","Data":"5cd8bcebbd6a9afc0b80db1a3f541ea12dca840b915283d7feff1dc40dfe5831"} Oct 30 06:57:25.068741 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.068677 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cd8bcebbd6a9afc0b80db1a3f541ea12dca840b915283d7feff1dc40dfe5831" Oct 30 06:57:25.075704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.075675 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" event={"ID":"9702025a-0ba2-4515-9118-d700256f7aa8","Type":"ContainerStarted","Data":"49e62e236806b425ace276afd8a64f3c72e19ecb06d06581407b8ff10dbeb9b0"} Oct 30 06:57:25.217992 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.217939 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:25.318813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.318780 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-xks-ca\") pod \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " Oct 30 06:57:25.318980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.318912 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-openshift-ca\") pod \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " Oct 30 06:57:25.318980 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.318956 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-kube-api-access-c6qw7\" (UniqueName: \"kubernetes.io/projected/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-kube-api-access-c6qw7\") pod \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\" (UID: \"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd\") " Oct 30 06:57:25.319266 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.319243 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-openshift-ca" pod "d9b9c3f0-9eda-481c-bf6a-86f72cb837fd" (UID: "d9b9c3f0-9eda-481c-bf6a-86f72cb837fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:25.319554 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.319532 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:25.321024 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.320998 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-xks-ca" pod "d9b9c3f0-9eda-481c-bf6a-86f72cb837fd" (UID: "d9b9c3f0-9eda-481c-bf6a-86f72cb837fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:25.321265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.321243 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-kube-api-access-c6qw7" pod "d9b9c3f0-9eda-481c-bf6a-86f72cb837fd" (UID: "d9b9c3f0-9eda-481c-bf6a-86f72cb837fd"). InnerVolumeSpecName "kube-api-access-c6qw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:25.421073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.421040 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6qw7\" (UniqueName: \"kubernetes.io/projected/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-kube-api-access-c6qw7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:25.421073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:25.421069 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:26.088626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.088596 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" Oct 30 06:57:26.089077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.088621 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8" event={"ID":"d9b9c3f0-9eda-481c-bf6a-86f72cb837fd","Type":"ContainerDied","Data":"581d055e6f9d0e64a72bdb3f28f6009ed2f1871c25e5e33542fca49041e83353"} Oct 30 06:57:26.089077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.088682 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581d055e6f9d0e64a72bdb3f28f6009ed2f1871c25e5e33542fca49041e83353" Oct 30 06:57:26.090850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.090821 5001 generic.go:356] "Generic (PLEG): container finished" podID="9702025a-0ba2-4515-9118-d700256f7aa8" containerID="1d538fea4fadf7fa582614744cb5cc589c527a42edcb1f0b46dff9d3bcdfb38c" exitCode=0 Oct 30 06:57:26.090999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.090892 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" event={"ID":"9702025a-0ba2-4515-9118-d700256f7aa8","Type":"ContainerDied","Data":"1d538fea4fadf7fa582614744cb5cc589c527a42edcb1f0b46dff9d3bcdfb38c"} Oct 30 06:57:26.363095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.363068 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:57:26.402900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.402857 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:57:26.534661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.534632 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c77805f-d7de-44d8-8a27-3c910cb60b4e-kube-api-access-6gwtt\" (UniqueName: \"kubernetes.io/projected/1c77805f-d7de-44d8-8a27-3c910cb60b4e-kube-api-access-6gwtt\") pod \"1c77805f-d7de-44d8-8a27-3c910cb60b4e\" (UID: \"1c77805f-d7de-44d8-8a27-3c910cb60b4e\") " Oct 30 06:57:26.534813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.534775 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c77805f-d7de-44d8-8a27-3c910cb60b4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c77805f-d7de-44d8-8a27-3c910cb60b4e-plugins-config-volume\") pod \"1c77805f-d7de-44d8-8a27-3c910cb60b4e\" (UID: \"1c77805f-d7de-44d8-8a27-3c910cb60b4e\") " Oct 30 06:57:26.534881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.534835 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/791a33a9-9513-4bd6-bd64-0c37766184df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791a33a9-9513-4bd6-bd64-0c37766184df-plugins-config-volume\") pod \"791a33a9-9513-4bd6-bd64-0c37766184df\" (UID: \"791a33a9-9513-4bd6-bd64-0c37766184df\") " Oct 30 06:57:26.535109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.535083 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/791a33a9-9513-4bd6-bd64-0c37766184df-kube-api-access-jdrfv\" (UniqueName: \"kubernetes.io/projected/791a33a9-9513-4bd6-bd64-0c37766184df-kube-api-access-jdrfv\") pod \"791a33a9-9513-4bd6-bd64-0c37766184df\" (UID: \"791a33a9-9513-4bd6-bd64-0c37766184df\") " Oct 30 06:57:26.535253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.535199 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c77805f-d7de-44d8-8a27-3c910cb60b4e-plugins-config-volume" pod "1c77805f-d7de-44d8-8a27-3c910cb60b4e" (UID: "1c77805f-d7de-44d8-8a27-3c910cb60b4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:26.535253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.535201 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791a33a9-9513-4bd6-bd64-0c37766184df-plugins-config-volume" pod "791a33a9-9513-4bd6-bd64-0c37766184df" (UID: "791a33a9-9513-4bd6-bd64-0c37766184df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:26.535641 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.535620 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c77805f-d7de-44d8-8a27-3c910cb60b4e-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:26.535716 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.535647 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791a33a9-9513-4bd6-bd64-0c37766184df-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:26.537153 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.537130 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c77805f-d7de-44d8-8a27-3c910cb60b4e-kube-api-access-6gwtt" pod "1c77805f-d7de-44d8-8a27-3c910cb60b4e" (UID: "1c77805f-d7de-44d8-8a27-3c910cb60b4e"). InnerVolumeSpecName "kube-api-access-6gwtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:26.537430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.537410 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791a33a9-9513-4bd6-bd64-0c37766184df-kube-api-access-jdrfv" pod "791a33a9-9513-4bd6-bd64-0c37766184df" (UID: "791a33a9-9513-4bd6-bd64-0c37766184df"). InnerVolumeSpecName "kube-api-access-jdrfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:26.637123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.637034 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gwtt\" (UniqueName: \"kubernetes.io/projected/1c77805f-d7de-44d8-8a27-3c910cb60b4e-kube-api-access-6gwtt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:26.637123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:26.637066 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdrfv\" (UniqueName: \"kubernetes.io/projected/791a33a9-9513-4bd6-bd64-0c37766184df-kube-api-access-jdrfv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:27.039625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.039598 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:57:27.102893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.102860 5001 generic.go:356] "Generic (PLEG): container finished" podID="791a33a9-9513-4bd6-bd64-0c37766184df" containerID="ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40" exitCode=0 Oct 30 06:57:27.103468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.102917 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" Oct 30 06:57:27.103468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.102955 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" event={"ID":"791a33a9-9513-4bd6-bd64-0c37766184df","Type":"ContainerDied","Data":"ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40"} Oct 30 06:57:27.103468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.103029 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6" event={"ID":"791a33a9-9513-4bd6-bd64-0c37766184df","Type":"ContainerDied","Data":"65e31f74be11be3bdec3ae855b0a1f4176b127daa78220594b8f520131084582"} Oct 30 06:57:27.103468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.103063 5001 scope.go:122] "RemoveContainer" containerID="ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40" Oct 30 06:57:27.105635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.105610 5001 generic.go:356] "Generic (PLEG): container finished" podID="f616e0d8-f098-4e17-8dfc-ce9076813eb0" containerID="11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe" exitCode=0 Oct 30 06:57:27.105840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.105827 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" Oct 30 06:57:27.105958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.105944 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" event={"ID":"f616e0d8-f098-4e17-8dfc-ce9076813eb0","Type":"ContainerDied","Data":"11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe"} Oct 30 06:57:27.106029 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.105974 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps" event={"ID":"f616e0d8-f098-4e17-8dfc-ce9076813eb0","Type":"ContainerDied","Data":"4b1ad848d3dfc3522cc0080b251ec0f32f74bd61424e14b28df42721d71c261b"} Oct 30 06:57:27.107578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.107548 5001 generic.go:356] "Generic (PLEG): container finished" podID="1c77805f-d7de-44d8-8a27-3c910cb60b4e" containerID="ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721" exitCode=0 Oct 30 06:57:27.107697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.107577 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" event={"ID":"1c77805f-d7de-44d8-8a27-3c910cb60b4e","Type":"ContainerDied","Data":"ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721"} Oct 30 06:57:27.107697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.107598 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" Oct 30 06:57:27.107697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.107628 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd" event={"ID":"1c77805f-d7de-44d8-8a27-3c910cb60b4e","Type":"ContainerDied","Data":"aad76a0677452648dc29adc50cca7a29ae5fac80a19a6fca85210d318c420e3c"} Oct 30 06:57:27.130040 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.130011 5001 scope.go:122] "RemoveContainer" containerID="ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40" Oct 30 06:57:27.130540 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:27.130507 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40\": container with ID starting with ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40 not found: ID does not exist" containerID="ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40" Oct 30 06:57:27.130670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.130551 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40"} err="failed to get container status \"ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40\": rpc error: code = NotFound desc = could not find container \"ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40\": container with ID starting with ffe38e10560f9cd6fa68543ed4167150bd15faf1c3e74823625698ad33afef40 not found: ID does not exist" Oct 30 06:57:27.130670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.130576 5001 scope.go:122] "RemoveContainer" containerID="11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe" Oct 30 06:57:27.144362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.144331 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f616e0d8-f098-4e17-8dfc-ce9076813eb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f616e0d8-f098-4e17-8dfc-ce9076813eb0-plugins-config-volume\") pod \"f616e0d8-f098-4e17-8dfc-ce9076813eb0\" (UID: \"f616e0d8-f098-4e17-8dfc-ce9076813eb0\") " Oct 30 06:57:27.144585 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.144569 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f616e0d8-f098-4e17-8dfc-ce9076813eb0-kube-api-access-k52wq\" (UniqueName: \"kubernetes.io/projected/f616e0d8-f098-4e17-8dfc-ce9076813eb0-kube-api-access-k52wq\") pod \"f616e0d8-f098-4e17-8dfc-ce9076813eb0\" (UID: \"f616e0d8-f098-4e17-8dfc-ce9076813eb0\") " Oct 30 06:57:27.146578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.146522 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f616e0d8-f098-4e17-8dfc-ce9076813eb0-plugins-config-volume" pod "f616e0d8-f098-4e17-8dfc-ce9076813eb0" (UID: "f616e0d8-f098-4e17-8dfc-ce9076813eb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:27.148645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.148582 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f616e0d8-f098-4e17-8dfc-ce9076813eb0-kube-api-access-k52wq" pod "f616e0d8-f098-4e17-8dfc-ce9076813eb0" (UID: "f616e0d8-f098-4e17-8dfc-ce9076813eb0"). InnerVolumeSpecName "kube-api-access-k52wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:27.152960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.152915 5001 scope.go:122] "RemoveContainer" containerID="11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe" Oct 30 06:57:27.153383 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:27.153348 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe\": container with ID starting with 11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe not found: ID does not exist" containerID="11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe" Oct 30 06:57:27.153494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.153385 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe"} err="failed to get container status \"11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe\": rpc error: code = NotFound desc = could not find container \"11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe\": container with ID starting with 11c739c74b4fd073ccd25949d78e6ad14152d0f401e1b2073ca13f976905d4fe not found: ID does not exist" Oct 30 06:57:27.153494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.153409 5001 scope.go:122] "RemoveContainer" containerID="ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721" Oct 30 06:57:27.166070 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.166047 5001 scope.go:122] "RemoveContainer" containerID="ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721" Oct 30 06:57:27.166436 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:27.166400 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721\": container with ID starting with ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721 not found: ID does not exist" containerID="ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721" Oct 30 06:57:27.166514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.166447 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721"} err="failed to get container status \"ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721\": rpc error: code = NotFound desc = could not find container \"ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721\": container with ID starting with ca141843df5295e0ecd55229be3e4ab56d74b9882c1e763535f4c4baace09721 not found: ID does not exist" Oct 30 06:57:27.192634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.192329 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6"] Oct 30 06:57:27.203389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.203360 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4f3973-768c84dbc-48kw6"] Oct 30 06:57:27.231090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.231059 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd"] Oct 30 06:57:27.239280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.239254 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4f3973-76bb9cf8b8-47czd"] Oct 30 06:57:27.247167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.247078 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f616e0d8-f098-4e17-8dfc-ce9076813eb0-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:27.247167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.247105 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k52wq\" (UniqueName: \"kubernetes.io/projected/f616e0d8-f098-4e17-8dfc-ce9076813eb0-kube-api-access-k52wq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:27.312468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.312442 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:27.449634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.449596 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-xks-ca\") pod \"9702025a-0ba2-4515-9118-d700256f7aa8\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " Oct 30 06:57:27.449802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.449765 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9702025a-0ba2-4515-9118-d700256f7aa8-kube-api-access-lcdw7\" (UniqueName: \"kubernetes.io/projected/9702025a-0ba2-4515-9118-d700256f7aa8-kube-api-access-lcdw7\") pod \"9702025a-0ba2-4515-9118-d700256f7aa8\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " Oct 30 06:57:27.449912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.449898 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-openshift-ca\") pod \"9702025a-0ba2-4515-9118-d700256f7aa8\" (UID: \"9702025a-0ba2-4515-9118-d700256f7aa8\") " Oct 30 06:57:27.450537 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.450413 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-openshift-ca" pod "9702025a-0ba2-4515-9118-d700256f7aa8" (UID: "9702025a-0ba2-4515-9118-d700256f7aa8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:27.453708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.453684 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-xks-ca" pod "9702025a-0ba2-4515-9118-d700256f7aa8" (UID: "9702025a-0ba2-4515-9118-d700256f7aa8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:27.453842 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.453790 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9702025a-0ba2-4515-9118-d700256f7aa8-kube-api-access-lcdw7" pod "9702025a-0ba2-4515-9118-d700256f7aa8" (UID: "9702025a-0ba2-4515-9118-d700256f7aa8"). InnerVolumeSpecName "kube-api-access-lcdw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:27.457067 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.457043 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps"] Oct 30 06:57:27.464816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.464797 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b22a13-59df5589d7-7xtps"] Oct 30 06:57:27.551978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.551941 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:27.551978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.551972 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9702025a-0ba2-4515-9118-d700256f7aa8-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:27.551978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:27.551982 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcdw7\" (UniqueName: \"kubernetes.io/projected/9702025a-0ba2-4515-9118-d700256f7aa8-kube-api-access-lcdw7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.120931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.120897 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c77805f-d7de-44d8-8a27-3c910cb60b4e" path="/var/lib/kubelet/pods/1c77805f-d7de-44d8-8a27-3c910cb60b4e/volumes" Oct 30 06:57:28.121510 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.121494 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="791a33a9-9513-4bd6-bd64-0c37766184df" path="/var/lib/kubelet/pods/791a33a9-9513-4bd6-bd64-0c37766184df/volumes" Oct 30 06:57:28.121961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.121946 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f616e0d8-f098-4e17-8dfc-ce9076813eb0" path="/var/lib/kubelet/pods/f616e0d8-f098-4e17-8dfc-ce9076813eb0/volumes" Oct 30 06:57:28.123361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.123336 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" event={"ID":"9702025a-0ba2-4515-9118-d700256f7aa8","Type":"ContainerDied","Data":"49e62e236806b425ace276afd8a64f3c72e19ecb06d06581407b8ff10dbeb9b0"} Oct 30 06:57:28.123453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.123381 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e62e236806b425ace276afd8a64f3c72e19ecb06d06581407b8ff10dbeb9b0" Oct 30 06:57:28.123670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.123654 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr" Oct 30 06:57:28.708278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.708255 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:57:28.880443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.880405 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-socket\") pod \"06bbf981-0119-4ab2-9a8d-51e21306a98d\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " Oct 30 06:57:28.880660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.880507 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-token\" (UniqueName: \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-token\") pod \"06bbf981-0119-4ab2-9a8d-51e21306a98d\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " Oct 30 06:57:28.880660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.880540 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-credential-socket\") pod \"06bbf981-0119-4ab2-9a8d-51e21306a98d\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " Oct 30 06:57:28.880780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.880690 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-data\") pod \"06bbf981-0119-4ab2-9a8d-51e21306a98d\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " Oct 30 06:57:28.880780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.880758 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-socket" pod "06bbf981-0119-4ab2-9a8d-51e21306a98d" (UID: "06bbf981-0119-4ab2-9a8d-51e21306a98d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:28.880901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.880814 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-kube-api-access-dh6k4\" (UniqueName: \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-kube-api-access-dh6k4\") pod \"06bbf981-0119-4ab2-9a8d-51e21306a98d\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " Oct 30 06:57:28.880901 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.880854 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06bbf981-0119-4ab2-9a8d-51e21306a98d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bbf981-0119-4ab2-9a8d-51e21306a98d-istiod-ca-cert\") pod \"06bbf981-0119-4ab2-9a8d-51e21306a98d\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " Oct 30 06:57:28.881013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.880899 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-podinfo\") pod \"06bbf981-0119-4ab2-9a8d-51e21306a98d\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " Oct 30 06:57:28.881013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.880963 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-certs\") pod \"06bbf981-0119-4ab2-9a8d-51e21306a98d\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " Oct 30 06:57:28.881013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.880969 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-credential-socket" pod "06bbf981-0119-4ab2-9a8d-51e21306a98d" (UID: "06bbf981-0119-4ab2-9a8d-51e21306a98d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:28.881198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.881051 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-envoy\") pod \"06bbf981-0119-4ab2-9a8d-51e21306a98d\" (UID: \"06bbf981-0119-4ab2-9a8d-51e21306a98d\") " Oct 30 06:57:28.881262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.881204 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-data" pod "06bbf981-0119-4ab2-9a8d-51e21306a98d" (UID: "06bbf981-0119-4ab2-9a8d-51e21306a98d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:28.881367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.881341 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bbf981-0119-4ab2-9a8d-51e21306a98d-istiod-ca-cert" pod "06bbf981-0119-4ab2-9a8d-51e21306a98d" (UID: "06bbf981-0119-4ab2-9a8d-51e21306a98d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:28.881367 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.881349 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-certs" pod "06bbf981-0119-4ab2-9a8d-51e21306a98d" (UID: "06bbf981-0119-4ab2-9a8d-51e21306a98d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:28.881997 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.881968 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.882121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.882011 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.882121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.882028 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bbf981-0119-4ab2-9a8d-51e21306a98d-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.882121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.882085 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.882121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.882100 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.883746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.883719 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-kube-api-access-dh6k4" pod "06bbf981-0119-4ab2-9a8d-51e21306a98d" (UID: "06bbf981-0119-4ab2-9a8d-51e21306a98d"). InnerVolumeSpecName "kube-api-access-dh6k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:28.883862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.883846 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-podinfo" pod "06bbf981-0119-4ab2-9a8d-51e21306a98d" (UID: "06bbf981-0119-4ab2-9a8d-51e21306a98d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:57:28.884094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.884074 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-envoy" pod "06bbf981-0119-4ab2-9a8d-51e21306a98d" (UID: "06bbf981-0119-4ab2-9a8d-51e21306a98d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:28.884433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.884409 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-token" pod "06bbf981-0119-4ab2-9a8d-51e21306a98d" (UID: "06bbf981-0119-4ab2-9a8d-51e21306a98d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:28.983183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.983096 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.983183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.983127 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh6k4\" (UniqueName: \"kubernetes.io/projected/06bbf981-0119-4ab2-9a8d-51e21306a98d-kube-api-access-dh6k4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.983183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.983140 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.983183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:28.983149 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bbf981-0119-4ab2-9a8d-51e21306a98d-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:29.135352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.135292 5001 generic.go:356] "Generic (PLEG): container finished" podID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerID="33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77" exitCode=0 Oct 30 06:57:29.135352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.135349 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" event={"ID":"06bbf981-0119-4ab2-9a8d-51e21306a98d","Type":"ContainerDied","Data":"33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77"} Oct 30 06:57:29.135831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.135405 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" event={"ID":"06bbf981-0119-4ab2-9a8d-51e21306a98d","Type":"ContainerDied","Data":"a76f710c353376bb86787186530fdc2775dfa473c74d399de4821b44cf2d47e0"} Oct 30 06:57:29.135831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.135431 5001 scope.go:122] "RemoveContainer" containerID="33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77" Oct 30 06:57:29.135831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.135357 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv" Oct 30 06:57:29.148227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.148209 5001 scope.go:122] "RemoveContainer" containerID="f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df" Oct 30 06:57:29.159777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.159754 5001 scope.go:122] "RemoveContainer" containerID="33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77" Oct 30 06:57:29.160091 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:29.160066 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77\": container with ID starting with 33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77 not found: ID does not exist" containerID="33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77" Oct 30 06:57:29.160171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.160104 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77"} err="failed to get container status \"33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77\": rpc error: code = NotFound desc = could not find container \"33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77\": container with ID starting with 33344eb6dca6c5ff1e0d2aa0f574e9548d3edebc04e6421d2fd7ab3b8586db77 not found: ID does not exist" Oct 30 06:57:29.160171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.160128 5001 scope.go:122] "RemoveContainer" containerID="f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df" Oct 30 06:57:29.160435 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:29.160415 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df\": container with ID starting with f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df not found: ID does not exist" containerID="f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df" Oct 30 06:57:29.160503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.160441 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df"} err="failed to get container status \"f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df\": rpc error: code = NotFound desc = could not find container \"f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df\": container with ID starting with f29a350384e180e17a9ad2b09ccead22089f88cd57b9af05e5846c02120d78df not found: ID does not exist" Oct 30 06:57:29.189708 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.189674 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv"] Oct 30 06:57:29.194272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:29.194244 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-z6dnv"] Oct 30 06:57:30.120371 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:30.120339 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" path="/var/lib/kubelet/pods/06bbf981-0119-4ab2-9a8d-51e21306a98d/volumes" Oct 30 06:57:30.148566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:30.148531 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2"] Oct 30 06:57:30.180532 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:30.180497 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf"] Oct 30 06:57:30.904352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:30.900792 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9"] Oct 30 06:57:30.926883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:30.926847 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh"] Oct 30 06:57:32.506098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:32.506062 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr"] Oct 30 06:57:32.545065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:32.545031 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb"] Oct 30 06:57:33.332174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:33.332136 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm"] Oct 30 06:57:33.344575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:33.344130 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-b9ff93-hz6wm"] Oct 30 06:57:34.119341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:34.119278 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f0337b-5822-4a2f-83fc-9bc5c229124b" path="/var/lib/kubelet/pods/f9f0337b-5822-4a2f-83fc-9bc5c229124b/volumes" Oct 30 06:57:35.149388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:35.149342 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" podUID="d781ce9b-5bf3-4ea9-870b-b22a77d02929" containerName="payload-processing" containerID="cri-o://102fef86ee7a0a308e7978dbc2de26e64539b0e9cd5318575dbefa75b70d60d0" gracePeriod=25 Oct 30 06:57:35.181445 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:35.181392 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" podUID="05977063-0120-460c-bee1-8ae684ea6ac7" containerName="payload-pre-processing" containerID="cri-o://59343c3b3a13fb39f6703363d8c4181383d4934c5d5df6da989cc46fd303e8e3" gracePeriod=25 Oct 30 06:57:35.822104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:35.822062 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" podUID="05977063-0120-460c-bee1-8ae684ea6ac7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:35.822291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:35.822126 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" podUID="d781ce9b-5bf3-4ea9-870b-b22a77d02929" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:35.901604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:35.901567 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" podUID="fc4ef9dd-2519-4506-a03e-3205ad65d331" containerName="payload-processing" containerID="cri-o://ad9506eaf4662347ba1dcff4533f3198fe58863b1649c9dba63b44dc754acc29" gracePeriod=25 Oct 30 06:57:35.927461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:35.927424 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" podUID="635a07fa-78b2-457e-9f7e-f41d6610b469" containerName="payload-pre-processing" containerID="cri-o://fe7cb610461fd97d6aaf39540832119ab0fe1db5acf1996bda71b54acfc85dc0" gracePeriod=25 Oct 30 06:57:36.799979 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:36.799938 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" podUID="f54e188f-35ed-4d3a-b66a-9ed12b084c36" containerName="main" containerID="cri-o://d57154351f35b300dd76fdfa5283289588d6de04e500ad002d27e327a6cc8d5d" gracePeriod=45 Oct 30 06:57:37.236730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.236595 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" podUID="635a07fa-78b2-457e-9f7e-f41d6610b469" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:37.255495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.251943 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" podUID="fc4ef9dd-2519-4506-a03e-3205ad65d331" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:37.271897 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.271864 5001 generic.go:356] "Generic (PLEG): container finished" podID="f54e188f-35ed-4d3a-b66a-9ed12b084c36" containerID="d57154351f35b300dd76fdfa5283289588d6de04e500ad002d27e327a6cc8d5d" exitCode=0 Oct 30 06:57:37.272100 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.271922 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" event={"ID":"f54e188f-35ed-4d3a-b66a-9ed12b084c36","Type":"ContainerDied","Data":"d57154351f35b300dd76fdfa5283289588d6de04e500ad002d27e327a6cc8d5d"} Oct 30 06:57:37.300568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.300533 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:57:37.427214 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.427176 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw"] Oct 30 06:57:37.432372 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.432348 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62efb8-swzrw"] Oct 30 06:57:37.441795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.441753 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-dshm\") pod \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " Oct 30 06:57:37.441949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.441868 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tls-certs\" (UniqueName: \"kubernetes.io/secret/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tls-certs\") pod \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " Oct 30 06:57:37.441949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.441892 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-model-cache\") pod \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " Oct 30 06:57:37.442080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.441969 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54e188f-35ed-4d3a-b66a-9ed12b084c36-kube-api-access-sjdx9\" (UniqueName: \"kubernetes.io/projected/f54e188f-35ed-4d3a-b66a-9ed12b084c36-kube-api-access-sjdx9\") pod \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " Oct 30 06:57:37.442080 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.442058 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-home\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-home\") pod \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " Oct 30 06:57:37.442183 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.442088 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tmp-dir\") pod \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\" (UID: \"f54e188f-35ed-4d3a-b66a-9ed12b084c36\") " Oct 30 06:57:37.442246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.442204 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-model-cache" pod "f54e188f-35ed-4d3a-b66a-9ed12b084c36" (UID: "f54e188f-35ed-4d3a-b66a-9ed12b084c36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:37.442568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.442423 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-home" pod "f54e188f-35ed-4d3a-b66a-9ed12b084c36" (UID: "f54e188f-35ed-4d3a-b66a-9ed12b084c36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:37.442727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.442693 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tmp-dir" pod "f54e188f-35ed-4d3a-b66a-9ed12b084c36" (UID: "f54e188f-35ed-4d3a-b66a-9ed12b084c36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:37.443020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.442998 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.443097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.443027 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.443097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.443040 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.444735 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.444703 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-dshm" pod "f54e188f-35ed-4d3a-b66a-9ed12b084c36" (UID: "f54e188f-35ed-4d3a-b66a-9ed12b084c36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:37.487281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.487091 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54e188f-35ed-4d3a-b66a-9ed12b084c36-kube-api-access-sjdx9" pod "f54e188f-35ed-4d3a-b66a-9ed12b084c36" (UID: "f54e188f-35ed-4d3a-b66a-9ed12b084c36"). InnerVolumeSpecName "kube-api-access-sjdx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:37.487670 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.487633 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tls-certs" pod "f54e188f-35ed-4d3a-b66a-9ed12b084c36" (UID: "f54e188f-35ed-4d3a-b66a-9ed12b084c36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:37.507346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.507313 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" podUID="acd94f6d-738f-46ea-8974-084320983b78" containerName="payload-processing" containerID="cri-o://f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8" gracePeriod=25 Oct 30 06:57:37.544117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.544091 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54e188f-35ed-4d3a-b66a-9ed12b084c36-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.544117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.544119 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54e188f-35ed-4d3a-b66a-9ed12b084c36-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.544295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.544128 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjdx9\" (UniqueName: \"kubernetes.io/projected/f54e188f-35ed-4d3a-b66a-9ed12b084c36-kube-api-access-sjdx9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.545469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.545414 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" podUID="067fb315-b20e-42f3-b792-d081d921b529" containerName="payload-pre-processing" containerID="cri-o://753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575" gracePeriod=25 Oct 30 06:57:37.779042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.779014 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:57:37.950978 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.950901 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd94f6d-738f-46ea-8974-084320983b78-kube-api-access-vqvpv\" (UniqueName: \"kubernetes.io/projected/acd94f6d-738f-46ea-8974-084320983b78-kube-api-access-vqvpv\") pod \"acd94f6d-738f-46ea-8974-084320983b78\" (UID: \"acd94f6d-738f-46ea-8974-084320983b78\") " Oct 30 06:57:37.951427 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.951090 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd94f6d-738f-46ea-8974-084320983b78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acd94f6d-738f-46ea-8974-084320983b78-plugins-config-volume\") pod \"acd94f6d-738f-46ea-8974-084320983b78\" (UID: \"acd94f6d-738f-46ea-8974-084320983b78\") " Oct 30 06:57:37.951577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.951554 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd94f6d-738f-46ea-8974-084320983b78-plugins-config-volume" pod "acd94f6d-738f-46ea-8974-084320983b78" (UID: "acd94f6d-738f-46ea-8974-084320983b78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:37.953254 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:37.953232 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd94f6d-738f-46ea-8974-084320983b78-kube-api-access-vqvpv" pod "acd94f6d-738f-46ea-8974-084320983b78" (UID: "acd94f6d-738f-46ea-8974-084320983b78"). InnerVolumeSpecName "kube-api-access-vqvpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:38.053232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.053200 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acd94f6d-738f-46ea-8974-084320983b78-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:38.053232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.053229 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqvpv\" (UniqueName: \"kubernetes.io/projected/acd94f6d-738f-46ea-8974-084320983b78-kube-api-access-vqvpv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:38.121731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.121692 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae0e0d65-dc76-4fd9-ad98-9a77158f09f8" path="/var/lib/kubelet/pods/ae0e0d65-dc76-4fd9-ad98-9a77158f09f8/volumes" Oct 30 06:57:38.270602 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:38.270523 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:57:38.281960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.281922 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" event={"ID":"f54e188f-35ed-4d3a-b66a-9ed12b084c36","Type":"ContainerDied","Data":"4643279df88b2d9d1e6df1012c380b7f508421a2edc4f3ba8e03a395269890d7"} Oct 30 06:57:38.281960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.281951 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs" Oct 30 06:57:38.282198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.281988 5001 scope.go:122] "RemoveContainer" containerID="d57154351f35b300dd76fdfa5283289588d6de04e500ad002d27e327a6cc8d5d" Oct 30 06:57:38.284869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.284178 5001 generic.go:356] "Generic (PLEG): container finished" podID="acd94f6d-738f-46ea-8974-084320983b78" containerID="f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8" exitCode=0 Oct 30 06:57:38.284869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.284284 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" Oct 30 06:57:38.284869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.284451 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" event={"ID":"acd94f6d-738f-46ea-8974-084320983b78","Type":"ContainerDied","Data":"f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8"} Oct 30 06:57:38.284869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.284521 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr" event={"ID":"acd94f6d-738f-46ea-8974-084320983b78","Type":"ContainerDied","Data":"e6ccb60bdb0ee55b669785de20be808cf4474342ca4c4cfec1f1842aae3724aa"} Oct 30 06:57:38.309549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.309521 5001 scope.go:122] "RemoveContainer" containerID="f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8" Oct 30 06:57:38.321566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.321544 5001 scope.go:122] "RemoveContainer" containerID="f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8" Oct 30 06:57:38.321863 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:38.321845 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8\": container with ID starting with f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8 not found: ID does not exist" containerID="f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8" Oct 30 06:57:38.321934 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.321871 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8"} err="failed to get container status \"f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8\": rpc error: code = NotFound desc = could not find container \"f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8\": container with ID starting with f60fe2d744c55c3cdab58607ba972e6254509827a157488c456f68d9cd8b4fd8 not found: ID does not exist" Oct 30 06:57:38.337603 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.337573 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr"] Oct 30 06:57:38.350750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.350721 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8baf7e-5b76fb7786-mb5vr"] Oct 30 06:57:38.373512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.373481 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs"] Oct 30 06:57:38.381584 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:38.381555 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-be48b4/test-model-be48b4-kserve-855dffb897-jcnfs"] Oct 30 06:57:40.119660 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:40.119617 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd94f6d-738f-46ea-8974-084320983b78" path="/var/lib/kubelet/pods/acd94f6d-738f-46ea-8974-084320983b78/volumes" Oct 30 06:57:40.120360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:40.120338 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54e188f-35ed-4d3a-b66a-9ed12b084c36" path="/var/lib/kubelet/pods/f54e188f-35ed-4d3a-b66a-9ed12b084c36/volumes" Oct 30 06:57:41.707386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.707347 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8"] Oct 30 06:57:41.708066 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708051 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1c77805f-d7de-44d8-8a27-3c910cb60b4e" containerName="payload-processing" Oct 30 06:57:41.708113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708067 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c77805f-d7de-44d8-8a27-3c910cb60b4e" containerName="payload-processing" Oct 30 06:57:41.708113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708081 5001 cpu_manager.go:414] "containerMap: removing container" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" Oct 30 06:57:41.708113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708087 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" Oct 30 06:57:41.708113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708101 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ae0e0d65-dc76-4fd9-ad98-9a77158f09f8" containerName="revoke-keys" Oct 30 06:57:41.708113 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708108 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae0e0d65-dc76-4fd9-ad98-9a77158f09f8" containerName="revoke-keys" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708115 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f9f0337b-5822-4a2f-83fc-9bc5c229124b" containerName="revoke-keys" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708119 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f0337b-5822-4a2f-83fc-9bc5c229124b" containerName="revoke-keys" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708130 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d9b9c3f0-9eda-481c-bf6a-86f72cb837fd" containerName="revoke-keys" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708135 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b9c3f0-9eda-481c-bf6a-86f72cb837fd" containerName="revoke-keys" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708143 5001 cpu_manager.go:414] "containerMap: removing container" podUID="9702025a-0ba2-4515-9118-d700256f7aa8" containerName="revoke-keys" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708147 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9702025a-0ba2-4515-9118-d700256f7aa8" containerName="revoke-keys" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708161 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f616e0d8-f098-4e17-8dfc-ce9076813eb0" containerName="payload-pre-processing" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708166 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f616e0d8-f098-4e17-8dfc-ce9076813eb0" containerName="payload-pre-processing" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708182 5001 cpu_manager.go:414] "containerMap: removing container" podUID="acd94f6d-738f-46ea-8974-084320983b78" containerName="payload-processing" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708186 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd94f6d-738f-46ea-8974-084320983b78" containerName="payload-processing" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708192 5001 cpu_manager.go:414] "containerMap: removing container" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708196 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708207 5001 cpu_manager.go:414] "containerMap: removing container" podUID="791a33a9-9513-4bd6-bd64-0c37766184df" containerName="payload-pre-processing" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708211 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="791a33a9-9513-4bd6-bd64-0c37766184df" containerName="payload-pre-processing" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708223 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f54e188f-35ed-4d3a-b66a-9ed12b084c36" containerName="main" Oct 30 06:57:41.708277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708228 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54e188f-35ed-4d3a-b66a-9ed12b084c36" containerName="main" Oct 30 06:57:41.708800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708338 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f616e0d8-f098-4e17-8dfc-ce9076813eb0" containerName="payload-pre-processing" Oct 30 06:57:41.708800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708350 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd94f6d-738f-46ea-8974-084320983b78" containerName="payload-processing" Oct 30 06:57:41.708800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708357 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" Oct 30 06:57:41.708800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708363 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="791a33a9-9513-4bd6-bd64-0c37766184df" containerName="payload-pre-processing" Oct 30 06:57:41.708800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708372 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54e188f-35ed-4d3a-b66a-9ed12b084c36" containerName="main" Oct 30 06:57:41.708800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708380 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c77805f-d7de-44d8-8a27-3c910cb60b4e" containerName="payload-processing" Oct 30 06:57:41.708800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708388 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae0e0d65-dc76-4fd9-ad98-9a77158f09f8" containerName="revoke-keys" Oct 30 06:57:41.708800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708393 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f0337b-5822-4a2f-83fc-9bc5c229124b" containerName="revoke-keys" Oct 30 06:57:41.708800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708401 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b9c3f0-9eda-481c-bf6a-86f72cb837fd" containerName="revoke-keys" Oct 30 06:57:41.708800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.708405 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9702025a-0ba2-4515-9118-d700256f7aa8" containerName="revoke-keys" Oct 30 06:57:41.712127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.712111 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:41.732519 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.732488 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8"] Oct 30 06:57:41.737669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.737643 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:57:41.737669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.737648 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:57:41.737864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.737711 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:57:41.737864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.737746 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:57:41.807939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.807898 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:41.808144 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.808098 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j4nt\" (UniqueName: \"kubernetes.io/projected/a9b5f1a3-b8b4-4188-aef9-991a892e6995-kube-api-access-2j4nt\") pod \"maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:41.808226 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.808164 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:41.909441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.909405 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:41.909619 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.909548 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j4nt\" (UniqueName: \"kubernetes.io/projected/a9b5f1a3-b8b4-4188-aef9-991a892e6995-kube-api-access-2j4nt\") pod \"maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:41.909661 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.909621 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:41.910064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.910039 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:41.912231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.912199 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:41.932982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:41.932953 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j4nt\" (UniqueName: \"kubernetes.io/projected/a9b5f1a3-b8b4-4188-aef9-991a892e6995-kube-api-access-2j4nt\") pod \"maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:42.027127 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:42.027042 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:42.343949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:42.343924 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8"] Oct 30 06:57:42.345839 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:57:42.345811 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b5f1a3_b8b4_4188_aef9_991a892e6995.slice/crio-581d97e83f5aa2e8febfe518ae4c37200ab2477095aa3b764ef43e6e4eca5eef WatchSource:0}: Error finding container 581d97e83f5aa2e8febfe518ae4c37200ab2477095aa3b764ef43e6e4eca5eef: Status 404 returned error can't find the container with id 581d97e83f5aa2e8febfe518ae4c37200ab2477095aa3b764ef43e6e4eca5eef Oct 30 06:57:43.340993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:43.340951 5001 generic.go:356] "Generic (PLEG): container finished" podID="a9b5f1a3-b8b4-4188-aef9-991a892e6995" containerID="eb2ad80b452e5edbc1c9f373a1a8dddfc386b968f0ea1adf72b5027988e5e753" exitCode=0 Oct 30 06:57:43.341468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:43.341058 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" event={"ID":"a9b5f1a3-b8b4-4188-aef9-991a892e6995","Type":"ContainerDied","Data":"eb2ad80b452e5edbc1c9f373a1a8dddfc386b968f0ea1adf72b5027988e5e753"} Oct 30 06:57:43.341468 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:43.341110 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" event={"ID":"a9b5f1a3-b8b4-4188-aef9-991a892e6995","Type":"ContainerStarted","Data":"581d97e83f5aa2e8febfe518ae4c37200ab2477095aa3b764ef43e6e4eca5eef"} Oct 30 06:57:44.523569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.523543 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:44.545073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.545047 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-openshift-ca\") pod \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " Oct 30 06:57:44.545238 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.545225 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-xks-ca\") pod \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " Oct 30 06:57:44.545368 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.545354 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b5f1a3-b8b4-4188-aef9-991a892e6995-kube-api-access-2j4nt\" (UniqueName: \"kubernetes.io/projected/a9b5f1a3-b8b4-4188-aef9-991a892e6995-kube-api-access-2j4nt\") pod \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\" (UID: \"a9b5f1a3-b8b4-4188-aef9-991a892e6995\") " Oct 30 06:57:44.545413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.545378 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-openshift-ca" pod "a9b5f1a3-b8b4-4188-aef9-991a892e6995" (UID: "a9b5f1a3-b8b4-4188-aef9-991a892e6995"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:44.545876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.545852 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:44.547502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.547483 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-xks-ca" pod "a9b5f1a3-b8b4-4188-aef9-991a892e6995" (UID: "a9b5f1a3-b8b4-4188-aef9-991a892e6995"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:44.547591 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.547576 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b5f1a3-b8b4-4188-aef9-991a892e6995-kube-api-access-2j4nt" pod "a9b5f1a3-b8b4-4188-aef9-991a892e6995" (UID: "a9b5f1a3-b8b4-4188-aef9-991a892e6995"). InnerVolumeSpecName "kube-api-access-2j4nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:44.651076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.651039 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j4nt\" (UniqueName: \"kubernetes.io/projected/a9b5f1a3-b8b4-4188-aef9-991a892e6995-kube-api-access-2j4nt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:44.655171 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.651343 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b5f1a3-b8b4-4188-aef9-991a892e6995-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:44.902331 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.902232 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx"] Oct 30 06:57:44.902533 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:44.902476 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" containerID="cri-o://6c97f7ed9a4f9985c6d94e61802b535a2c777f871e4d10c2a3e58223849bf71d" gracePeriod=30 Oct 30 06:57:45.363543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.363509 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" Oct 30 06:57:45.363543 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.363529 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8" event={"ID":"a9b5f1a3-b8b4-4188-aef9-991a892e6995","Type":"ContainerDied","Data":"581d97e83f5aa2e8febfe518ae4c37200ab2477095aa3b764ef43e6e4eca5eef"} Oct 30 06:57:45.363788 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.363588 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581d97e83f5aa2e8febfe518ae4c37200ab2477095aa3b764ef43e6e4eca5eef" Oct 30 06:57:45.542341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.542293 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf"] Oct 30 06:57:45.543026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.543011 5001 cpu_manager.go:414] "containerMap: removing container" podUID="a9b5f1a3-b8b4-4188-aef9-991a892e6995" containerName="revoke-keys" Oct 30 06:57:45.543073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.543027 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b5f1a3-b8b4-4188-aef9-991a892e6995" containerName="revoke-keys" Oct 30 06:57:45.543174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.543165 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="06bbf981-0119-4ab2-9a8d-51e21306a98d" containerName="istio-proxy" Oct 30 06:57:45.543213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.543179 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b5f1a3-b8b4-4188-aef9-991a892e6995" containerName="revoke-keys" Oct 30 06:57:45.547041 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.547021 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:45.562450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.562406 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:57:45.562893 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.562869 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:57:45.569885 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.569863 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf"] Oct 30 06:57:45.584936 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.584913 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:45.585746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.585469 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdjk\" (UniqueName: \"kubernetes.io/projected/e2e7da2c-4f88-48b6-a09b-4c0371075b69-kube-api-access-btdjk\") pod \"maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:45.585746 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.585569 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:45.660590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.660503 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:57:45.660590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.660555 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:57:45.687148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.687107 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdjk\" (UniqueName: \"kubernetes.io/projected/e2e7da2c-4f88-48b6-a09b-4c0371075b69-kube-api-access-btdjk\") pod \"maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:45.687376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.687177 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:45.687376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.687285 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:45.687987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.687969 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:45.689831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.689809 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:45.710234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.710209 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdjk\" (UniqueName: \"kubernetes.io/projected/e2e7da2c-4f88-48b6-a09b-4c0371075b69-kube-api-access-btdjk\") pod \"maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:45.822122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.822089 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" podUID="d781ce9b-5bf3-4ea9-870b-b22a77d02929" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:45.822122 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.822105 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" podUID="05977063-0120-460c-bee1-8ae684ea6ac7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:45.866596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:45.866561 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:46.024277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:46.024242 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" podUID="067fb315-b20e-42f3-b792-d081d921b529" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:46.183657 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:46.183625 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf"] Oct 30 06:57:46.184703 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:57:46.184672 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e7da2c_4f88_48b6_a09b_4c0371075b69.slice/crio-be4f802f2844fd19a31aca3281ae4f12bfb413447e2ee215f14a5c689fc3dcf3 WatchSource:0}: Error finding container be4f802f2844fd19a31aca3281ae4f12bfb413447e2ee215f14a5c689fc3dcf3: Status 404 returned error can't find the container with id be4f802f2844fd19a31aca3281ae4f12bfb413447e2ee215f14a5c689fc3dcf3 Oct 30 06:57:46.375850 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:46.375811 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" event={"ID":"e2e7da2c-4f88-48b6-a09b-4c0371075b69","Type":"ContainerStarted","Data":"be4f802f2844fd19a31aca3281ae4f12bfb413447e2ee215f14a5c689fc3dcf3"} Oct 30 06:57:46.441664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:46.441573 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr"] Oct 30 06:57:47.131090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:47.131050 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" podUID="fc4ef9dd-2519-4506-a03e-3205ad65d331" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:47.131604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:47.131529 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" podUID="635a07fa-78b2-457e-9f7e-f41d6610b469" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:47.389133 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:47.389066 5001 generic.go:356] "Generic (PLEG): container finished" podID="e2e7da2c-4f88-48b6-a09b-4c0371075b69" containerID="a3ea756d9ca0b163f8a9348a1cf47b47797de56d4385dda9a863fd1764e226f8" exitCode=0 Oct 30 06:57:47.389278 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:47.389153 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" event={"ID":"e2e7da2c-4f88-48b6-a09b-4c0371075b69","Type":"ContainerDied","Data":"a3ea756d9ca0b163f8a9348a1cf47b47797de56d4385dda9a863fd1764e226f8"} Oct 30 06:57:48.589596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:48.589569 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:48.628256 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:48.628226 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-xks-ca\") pod \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " Oct 30 06:57:48.628598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:48.628567 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e7da2c-4f88-48b6-a09b-4c0371075b69-kube-api-access-btdjk\" (UniqueName: \"kubernetes.io/projected/e2e7da2c-4f88-48b6-a09b-4c0371075b69-kube-api-access-btdjk\") pod \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " Oct 30 06:57:48.628755 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:48.628698 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-openshift-ca\") pod \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\" (UID: \"e2e7da2c-4f88-48b6-a09b-4c0371075b69\") " Oct 30 06:57:48.629138 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:48.629098 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-openshift-ca" pod "e2e7da2c-4f88-48b6-a09b-4c0371075b69" (UID: "e2e7da2c-4f88-48b6-a09b-4c0371075b69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:48.629509 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:48.629484 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:48.631386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:48.631358 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-xks-ca" pod "e2e7da2c-4f88-48b6-a09b-4c0371075b69" (UID: "e2e7da2c-4f88-48b6-a09b-4c0371075b69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:48.631526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:48.631441 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e7da2c-4f88-48b6-a09b-4c0371075b69-kube-api-access-btdjk" pod "e2e7da2c-4f88-48b6-a09b-4c0371075b69" (UID: "e2e7da2c-4f88-48b6-a09b-4c0371075b69"). InnerVolumeSpecName "kube-api-access-btdjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:48.730713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:48.730678 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e7da2c-4f88-48b6-a09b-4c0371075b69-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:48.730713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:48.730711 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btdjk\" (UniqueName: \"kubernetes.io/projected/e2e7da2c-4f88-48b6-a09b-4c0371075b69-kube-api-access-btdjk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.364221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:49.364184 5001 patch_prober.go:29] interesting pod/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:57:49.364420 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:49.364241 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:57:49.408520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:49.408486 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" Oct 30 06:57:49.408704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:49.408528 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf" event={"ID":"e2e7da2c-4f88-48b6-a09b-4c0371075b69","Type":"ContainerDied","Data":"be4f802f2844fd19a31aca3281ae4f12bfb413447e2ee215f14a5c689fc3dcf3"} Oct 30 06:57:49.408704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:49.408574 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be4f802f2844fd19a31aca3281ae4f12bfb413447e2ee215f14a5c689fc3dcf3" Oct 30 06:57:50.420697 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.420665 5001 generic.go:356] "Generic (PLEG): container finished" podID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerID="6c97f7ed9a4f9985c6d94e61802b535a2c777f871e4d10c2a3e58223849bf71d" exitCode=0 Oct 30 06:57:50.421140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.420715 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" event={"ID":"0c7e9f40-f353-436a-b771-ea6223ac02db","Type":"ContainerDied","Data":"6c97f7ed9a4f9985c6d94e61802b535a2c777f871e4d10c2a3e58223849bf71d"} Oct 30 06:57:50.421140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.420774 5001 scope.go:122] "RemoveContainer" containerID="cab136d2978bfa28648e8d879fe56ef5fc787f4c97ff1794c610f8ece6d28496" Oct 30 06:57:50.612444 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.612418 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:57:50.656650 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.656621 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c7e9f40-f353-436a-b771-ea6223ac02db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7e9f40-f353-436a-b771-ea6223ac02db-istiod-ca-cert\") pod \"0c7e9f40-f353-436a-b771-ea6223ac02db\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " Oct 30 06:57:50.656827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.656660 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-credential-socket\") pod \"0c7e9f40-f353-436a-b771-ea6223ac02db\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " Oct 30 06:57:50.656827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.656713 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-kube-api-access-mz7h7\" (UniqueName: \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-kube-api-access-mz7h7\") pod \"0c7e9f40-f353-436a-b771-ea6223ac02db\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " Oct 30 06:57:50.656827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.656784 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-data\") pod \"0c7e9f40-f353-436a-b771-ea6223ac02db\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " Oct 30 06:57:50.656989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.656869 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-podinfo\") pod \"0c7e9f40-f353-436a-b771-ea6223ac02db\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " Oct 30 06:57:50.656989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.656904 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-envoy\") pod \"0c7e9f40-f353-436a-b771-ea6223ac02db\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " Oct 30 06:57:50.657105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.656993 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-token\" (UniqueName: \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-token\") pod \"0c7e9f40-f353-436a-b771-ea6223ac02db\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " Oct 30 06:57:50.657105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.657026 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-certs\") pod \"0c7e9f40-f353-436a-b771-ea6223ac02db\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " Oct 30 06:57:50.657105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.657075 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-socket\") pod \"0c7e9f40-f353-436a-b771-ea6223ac02db\" (UID: \"0c7e9f40-f353-436a-b771-ea6223ac02db\") " Oct 30 06:57:50.657105 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.657091 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7e9f40-f353-436a-b771-ea6223ac02db-istiod-ca-cert" pod "0c7e9f40-f353-436a-b771-ea6223ac02db" (UID: "0c7e9f40-f353-436a-b771-ea6223ac02db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:50.657383 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.657175 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-data" pod "0c7e9f40-f353-436a-b771-ea6223ac02db" (UID: "0c7e9f40-f353-436a-b771-ea6223ac02db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:50.657454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.657389 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-certs" pod "0c7e9f40-f353-436a-b771-ea6223ac02db" (UID: "0c7e9f40-f353-436a-b771-ea6223ac02db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:50.657624 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.657605 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-socket" pod "0c7e9f40-f353-436a-b771-ea6223ac02db" (UID: "0c7e9f40-f353-436a-b771-ea6223ac02db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:50.657802 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.657788 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7e9f40-f353-436a-b771-ea6223ac02db-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:50.657865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.657807 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:50.657865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.657821 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:50.657865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.657832 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:50.658031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.657861 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-credential-socket" pod "0c7e9f40-f353-436a-b771-ea6223ac02db" (UID: "0c7e9f40-f353-436a-b771-ea6223ac02db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:50.660345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.659938 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-kube-api-access-mz7h7" pod "0c7e9f40-f353-436a-b771-ea6223ac02db" (UID: "0c7e9f40-f353-436a-b771-ea6223ac02db"). InnerVolumeSpecName "kube-api-access-mz7h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:50.660345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.660016 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-envoy" pod "0c7e9f40-f353-436a-b771-ea6223ac02db" (UID: "0c7e9f40-f353-436a-b771-ea6223ac02db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:50.660345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.660196 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-podinfo" pod "0c7e9f40-f353-436a-b771-ea6223ac02db" (UID: "0c7e9f40-f353-436a-b771-ea6223ac02db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:57:50.661168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.661146 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-token" pod "0c7e9f40-f353-436a-b771-ea6223ac02db" (UID: "0c7e9f40-f353-436a-b771-ea6223ac02db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:50.760020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.759983 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:50.760020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.760020 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:50.760250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.760034 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:50.760250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.760046 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7e9f40-f353-436a-b771-ea6223ac02db-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:50.760250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:50.760060 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz7h7\" (UniqueName: \"kubernetes.io/projected/0c7e9f40-f353-436a-b771-ea6223ac02db-kube-api-access-mz7h7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:51.430747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:51.430719 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" Oct 30 06:57:51.431247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:51.430747 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx" event={"ID":"0c7e9f40-f353-436a-b771-ea6223ac02db","Type":"ContainerDied","Data":"b20d91be833f271be2aa11e3358a7e2889da0e927e015e4f4b1eb51bcf420764"} Oct 30 06:57:51.431247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:51.430811 5001 scope.go:122] "RemoveContainer" containerID="6c97f7ed9a4f9985c6d94e61802b535a2c777f871e4d10c2a3e58223849bf71d" Oct 30 06:57:51.497399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:51.497368 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx"] Oct 30 06:57:51.508276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:51.508248 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62efb8-openshift-default-848d569b9-f7dwx"] Oct 30 06:57:52.119373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:52.119338 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" path="/var/lib/kubelet/pods/0c7e9f40-f353-436a-b771-ea6223ac02db/volumes" Oct 30 06:57:52.202188 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:52.202149 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg"] Oct 30 06:57:55.489549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.486091 5001 generic.go:356] "Generic (PLEG): container finished" podID="d781ce9b-5bf3-4ea9-870b-b22a77d02929" containerID="102fef86ee7a0a308e7978dbc2de26e64539b0e9cd5318575dbefa75b70d60d0" exitCode=0 Oct 30 06:57:55.489549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.486365 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" event={"ID":"d781ce9b-5bf3-4ea9-870b-b22a77d02929","Type":"ContainerDied","Data":"102fef86ee7a0a308e7978dbc2de26e64539b0e9cd5318575dbefa75b70d60d0"} Oct 30 06:57:55.489549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.486426 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" event={"ID":"d781ce9b-5bf3-4ea9-870b-b22a77d02929","Type":"ContainerDied","Data":"868edd90db6467ef52a95a6ab955cec2df24ab6f939336f07d1d8e1593abeb1c"} Oct 30 06:57:55.489549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.486453 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="868edd90db6467ef52a95a6ab955cec2df24ab6f939336f07d1d8e1593abeb1c" Oct 30 06:57:55.489549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.488605 5001 generic.go:356] "Generic (PLEG): container finished" podID="05977063-0120-460c-bee1-8ae684ea6ac7" containerID="59343c3b3a13fb39f6703363d8c4181383d4934c5d5df6da989cc46fd303e8e3" exitCode=0 Oct 30 06:57:55.489549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.488665 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" event={"ID":"05977063-0120-460c-bee1-8ae684ea6ac7","Type":"ContainerDied","Data":"59343c3b3a13fb39f6703363d8c4181383d4934c5d5df6da989cc46fd303e8e3"} Oct 30 06:57:55.496187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.496162 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:57:55.525042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.525020 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:57:55.545211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.545182 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d781ce9b-5bf3-4ea9-870b-b22a77d02929-kube-api-access-jsft4\" (UniqueName: \"kubernetes.io/projected/d781ce9b-5bf3-4ea9-870b-b22a77d02929-kube-api-access-jsft4\") pod \"d781ce9b-5bf3-4ea9-870b-b22a77d02929\" (UID: \"d781ce9b-5bf3-4ea9-870b-b22a77d02929\") " Oct 30 06:57:55.545411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.545297 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d781ce9b-5bf3-4ea9-870b-b22a77d02929-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d781ce9b-5bf3-4ea9-870b-b22a77d02929-plugins-config-volume\") pod \"d781ce9b-5bf3-4ea9-870b-b22a77d02929\" (UID: \"d781ce9b-5bf3-4ea9-870b-b22a77d02929\") " Oct 30 06:57:55.545717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.545692 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d781ce9b-5bf3-4ea9-870b-b22a77d02929-plugins-config-volume" pod "d781ce9b-5bf3-4ea9-870b-b22a77d02929" (UID: "d781ce9b-5bf3-4ea9-870b-b22a77d02929"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:55.546021 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.546003 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d781ce9b-5bf3-4ea9-870b-b22a77d02929-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:55.547539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.547514 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d781ce9b-5bf3-4ea9-870b-b22a77d02929-kube-api-access-jsft4" pod "d781ce9b-5bf3-4ea9-870b-b22a77d02929" (UID: "d781ce9b-5bf3-4ea9-870b-b22a77d02929"). InnerVolumeSpecName "kube-api-access-jsft4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:55.646616 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.646584 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05977063-0120-460c-bee1-8ae684ea6ac7-kube-api-access-dzdck\" (UniqueName: \"kubernetes.io/projected/05977063-0120-460c-bee1-8ae684ea6ac7-kube-api-access-dzdck\") pod \"05977063-0120-460c-bee1-8ae684ea6ac7\" (UID: \"05977063-0120-460c-bee1-8ae684ea6ac7\") " Oct 30 06:57:55.646821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.646806 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05977063-0120-460c-bee1-8ae684ea6ac7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05977063-0120-460c-bee1-8ae684ea6ac7-plugins-config-volume\") pod \"05977063-0120-460c-bee1-8ae684ea6ac7\" (UID: \"05977063-0120-460c-bee1-8ae684ea6ac7\") " Oct 30 06:57:55.647177 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.647155 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05977063-0120-460c-bee1-8ae684ea6ac7-plugins-config-volume" pod "05977063-0120-460c-bee1-8ae684ea6ac7" (UID: "05977063-0120-460c-bee1-8ae684ea6ac7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:55.647237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.647228 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsft4\" (UniqueName: \"kubernetes.io/projected/d781ce9b-5bf3-4ea9-870b-b22a77d02929-kube-api-access-jsft4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:55.648870 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.648850 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05977063-0120-460c-bee1-8ae684ea6ac7-kube-api-access-dzdck" pod "05977063-0120-460c-bee1-8ae684ea6ac7" (UID: "05977063-0120-460c-bee1-8ae684ea6ac7"). InnerVolumeSpecName "kube-api-access-dzdck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:55.738521 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.738440 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv"] Oct 30 06:57:55.748561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.748535 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05977063-0120-460c-bee1-8ae684ea6ac7-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:55.748561 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:55.748560 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzdck\" (UniqueName: \"kubernetes.io/projected/05977063-0120-460c-bee1-8ae684ea6ac7-kube-api-access-dzdck\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:56.030140 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.030030 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" podUID="067fb315-b20e-42f3-b792-d081d921b529" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:56.501506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.501376 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8"] Oct 30 06:57:56.506664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.506639 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8baf7e-qmrf8"] Oct 30 06:57:56.521012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.520846 5001 generic.go:356] "Generic (PLEG): container finished" podID="635a07fa-78b2-457e-9f7e-f41d6610b469" containerID="fe7cb610461fd97d6aaf39540832119ab0fe1db5acf1996bda71b54acfc85dc0" exitCode=0 Oct 30 06:57:56.521012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.520946 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" event={"ID":"635a07fa-78b2-457e-9f7e-f41d6610b469","Type":"ContainerDied","Data":"fe7cb610461fd97d6aaf39540832119ab0fe1db5acf1996bda71b54acfc85dc0"} Oct 30 06:57:56.523604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.523580 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" event={"ID":"05977063-0120-460c-bee1-8ae684ea6ac7","Type":"ContainerDied","Data":"7d2c60b52227c23c6746898f118a475d709ece3c79a0147538ce8e3b59432b83"} Oct 30 06:57:56.523743 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.523636 5001 scope.go:122] "RemoveContainer" containerID="59343c3b3a13fb39f6703363d8c4181383d4934c5d5df6da989cc46fd303e8e3" Oct 30 06:57:56.524282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.524264 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf" Oct 30 06:57:56.530467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.530207 5001 generic.go:356] "Generic (PLEG): container finished" podID="fc4ef9dd-2519-4506-a03e-3205ad65d331" containerID="ad9506eaf4662347ba1dcff4533f3198fe58863b1649c9dba63b44dc754acc29" exitCode=0 Oct 30 06:57:56.531320 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.530633 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2" Oct 30 06:57:56.531320 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.531166 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" event={"ID":"fc4ef9dd-2519-4506-a03e-3205ad65d331","Type":"ContainerDied","Data":"ad9506eaf4662347ba1dcff4533f3198fe58863b1649c9dba63b44dc754acc29"} Oct 30 06:57:56.614827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.614801 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:57:56.659814 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.659787 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:57:56.669749 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.669700 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2"] Oct 30 06:57:56.675958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.675930 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62efb8-74d8fbfd66-2gbn2"] Oct 30 06:57:56.706888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.706863 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf"] Oct 30 06:57:56.711257 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.711234 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62efb8-5957d6896c-dvknf"] Oct 30 06:57:56.715552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.715532 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635a07fa-78b2-457e-9f7e-f41d6610b469-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635a07fa-78b2-457e-9f7e-f41d6610b469-plugins-config-volume\") pod \"635a07fa-78b2-457e-9f7e-f41d6610b469\" (UID: \"635a07fa-78b2-457e-9f7e-f41d6610b469\") " Oct 30 06:57:56.715711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.715698 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4ef9dd-2519-4506-a03e-3205ad65d331-kube-api-access-qw6g4\" (UniqueName: \"kubernetes.io/projected/fc4ef9dd-2519-4506-a03e-3205ad65d331-kube-api-access-qw6g4\") pod \"fc4ef9dd-2519-4506-a03e-3205ad65d331\" (UID: \"fc4ef9dd-2519-4506-a03e-3205ad65d331\") " Oct 30 06:57:56.715785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.715730 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4ef9dd-2519-4506-a03e-3205ad65d331-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4ef9dd-2519-4506-a03e-3205ad65d331-plugins-config-volume\") pod \"fc4ef9dd-2519-4506-a03e-3205ad65d331\" (UID: \"fc4ef9dd-2519-4506-a03e-3205ad65d331\") " Oct 30 06:57:56.715847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.715791 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635a07fa-78b2-457e-9f7e-f41d6610b469-kube-api-access-9ssdt\" (UniqueName: \"kubernetes.io/projected/635a07fa-78b2-457e-9f7e-f41d6610b469-kube-api-access-9ssdt\") pod \"635a07fa-78b2-457e-9f7e-f41d6610b469\" (UID: \"635a07fa-78b2-457e-9f7e-f41d6610b469\") " Oct 30 06:57:56.716087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.715964 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635a07fa-78b2-457e-9f7e-f41d6610b469-plugins-config-volume" pod "635a07fa-78b2-457e-9f7e-f41d6610b469" (UID: "635a07fa-78b2-457e-9f7e-f41d6610b469"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:56.716201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.716105 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4ef9dd-2519-4506-a03e-3205ad65d331-plugins-config-volume" pod "fc4ef9dd-2519-4506-a03e-3205ad65d331" (UID: "fc4ef9dd-2519-4506-a03e-3205ad65d331"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:56.716298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.716271 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4ef9dd-2519-4506-a03e-3205ad65d331-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:56.716298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.716288 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635a07fa-78b2-457e-9f7e-f41d6610b469-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:56.718063 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.718044 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4ef9dd-2519-4506-a03e-3205ad65d331-kube-api-access-qw6g4" pod "fc4ef9dd-2519-4506-a03e-3205ad65d331" (UID: "fc4ef9dd-2519-4506-a03e-3205ad65d331"). InnerVolumeSpecName "kube-api-access-qw6g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:56.718134 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.718086 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635a07fa-78b2-457e-9f7e-f41d6610b469-kube-api-access-9ssdt" pod "635a07fa-78b2-457e-9f7e-f41d6610b469" (UID: "635a07fa-78b2-457e-9f7e-f41d6610b469"). InnerVolumeSpecName "kube-api-access-9ssdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:56.818019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.817971 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw6g4\" (UniqueName: \"kubernetes.io/projected/fc4ef9dd-2519-4506-a03e-3205ad65d331-kube-api-access-qw6g4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:56.818019 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:56.818009 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ssdt\" (UniqueName: \"kubernetes.io/projected/635a07fa-78b2-457e-9f7e-f41d6610b469-kube-api-access-9ssdt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:57.203335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.203274 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" podUID="cb1e5e0a-f7a7-4a53-968a-3e8404fdce87" containerName="payload-pre-processing" containerID="cri-o://55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f" gracePeriod=25 Oct 30 06:57:57.483435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.483046 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:57:57.527589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.527558 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-kube-api-access-6qqr4\" (UniqueName: \"kubernetes.io/projected/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-kube-api-access-6qqr4\") pod \"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87\" (UID: \"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87\") " Oct 30 06:57:57.528031 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.527804 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-plugins-config-volume\") pod \"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87\" (UID: \"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87\") " Oct 30 06:57:57.528218 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.528195 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-plugins-config-volume" pod "cb1e5e0a-f7a7-4a53-968a-3e8404fdce87" (UID: "cb1e5e0a-f7a7-4a53-968a-3e8404fdce87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:57.528470 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.528451 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:57.529951 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.529928 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-kube-api-access-6qqr4" pod "cb1e5e0a-f7a7-4a53-968a-3e8404fdce87" (UID: "cb1e5e0a-f7a7-4a53-968a-3e8404fdce87"). InnerVolumeSpecName "kube-api-access-6qqr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:57.543943 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.543907 5001 generic.go:356] "Generic (PLEG): container finished" podID="cb1e5e0a-f7a7-4a53-968a-3e8404fdce87" containerID="55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f" exitCode=0 Oct 30 06:57:57.544072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.544009 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" event={"ID":"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87","Type":"ContainerDied","Data":"55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f"} Oct 30 06:57:57.544146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.544083 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" event={"ID":"cb1e5e0a-f7a7-4a53-968a-3e8404fdce87","Type":"ContainerDied","Data":"b1eb089f5a322bc1d9150bd2e770aa99cbc6ff7d7231432690b956b39f4a29ef"} Oct 30 06:57:57.544146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.544116 5001 scope.go:122] "RemoveContainer" containerID="55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f" Oct 30 06:57:57.544336 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.544271 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg" Oct 30 06:57:57.546577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.546553 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" event={"ID":"fc4ef9dd-2519-4506-a03e-3205ad65d331","Type":"ContainerDied","Data":"1490ea8960fd06883682878a48272061a7dee650ddd4921f5fa4cbf5af275f49"} Oct 30 06:57:57.546687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.546661 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9" Oct 30 06:57:57.557419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.553678 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" event={"ID":"635a07fa-78b2-457e-9f7e-f41d6610b469","Type":"ContainerDied","Data":"55904650a4f0064e57b82d8ce91fb278997e824c291a0b82cd8a7c64cee387ae"} Oct 30 06:57:57.557419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.553783 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh" Oct 30 06:57:57.561683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.561123 5001 scope.go:122] "RemoveContainer" containerID="55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f" Oct 30 06:57:57.561683 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:57.561441 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f\": container with ID starting with 55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f not found: ID does not exist" containerID="55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f" Oct 30 06:57:57.561683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.561476 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f"} err="failed to get container status \"55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f\": rpc error: code = NotFound desc = could not find container \"55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f\": container with ID starting with 55170ca7a651e7ed752a09e6d74d499da255f26c998042a3e30f6cec081dbf2f not found: ID does not exist" Oct 30 06:57:57.561683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.561499 5001 scope.go:122] "RemoveContainer" containerID="ad9506eaf4662347ba1dcff4533f3198fe58863b1649c9dba63b44dc754acc29" Oct 30 06:57:57.630239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.630210 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qqr4\" (UniqueName: \"kubernetes.io/projected/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87-kube-api-access-6qqr4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:57.635705 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.635675 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg"] Oct 30 06:57:57.640211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.640185 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a519e9-7b75656447-pgbsg"] Oct 30 06:57:57.663048 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.663017 5001 scope.go:122] "RemoveContainer" containerID="fe7cb610461fd97d6aaf39540832119ab0fe1db5acf1996bda71b54acfc85dc0" Oct 30 06:57:57.679375 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.679347 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9"] Oct 30 06:57:57.687193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.687154 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b9ff93-5b695fd5c6-w8ll9"] Oct 30 06:57:57.710189 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.710107 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh"] Oct 30 06:57:57.714535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.714491 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b9ff93-d89cd44bc-9lzbh"] Oct 30 06:57:57.822635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.822611 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:57:57.934877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.934838 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/067fb315-b20e-42f3-b792-d081d921b529-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067fb315-b20e-42f3-b792-d081d921b529-plugins-config-volume\") pod \"067fb315-b20e-42f3-b792-d081d921b529\" (UID: \"067fb315-b20e-42f3-b792-d081d921b529\") " Oct 30 06:57:57.935081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.934972 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/067fb315-b20e-42f3-b792-d081d921b529-kube-api-access-q5qhc\" (UniqueName: \"kubernetes.io/projected/067fb315-b20e-42f3-b792-d081d921b529-kube-api-access-q5qhc\") pod \"067fb315-b20e-42f3-b792-d081d921b529\" (UID: \"067fb315-b20e-42f3-b792-d081d921b529\") " Oct 30 06:57:57.935288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.935262 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067fb315-b20e-42f3-b792-d081d921b529-plugins-config-volume" pod "067fb315-b20e-42f3-b792-d081d921b529" (UID: "067fb315-b20e-42f3-b792-d081d921b529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:57.935618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.935600 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067fb315-b20e-42f3-b792-d081d921b529-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:57.937227 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:57.937206 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067fb315-b20e-42f3-b792-d081d921b529-kube-api-access-q5qhc" pod "067fb315-b20e-42f3-b792-d081d921b529" (UID: "067fb315-b20e-42f3-b792-d081d921b529"). InnerVolumeSpecName "kube-api-access-q5qhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:58.037245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.037157 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5qhc\" (UniqueName: \"kubernetes.io/projected/067fb315-b20e-42f3-b792-d081d921b529-kube-api-access-q5qhc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:57:58.118747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.118714 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05977063-0120-460c-bee1-8ae684ea6ac7" path="/var/lib/kubelet/pods/05977063-0120-460c-bee1-8ae684ea6ac7/volumes" Oct 30 06:57:58.119172 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.119157 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635a07fa-78b2-457e-9f7e-f41d6610b469" path="/var/lib/kubelet/pods/635a07fa-78b2-457e-9f7e-f41d6610b469/volumes" Oct 30 06:57:58.119608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.119594 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1e5e0a-f7a7-4a53-968a-3e8404fdce87" path="/var/lib/kubelet/pods/cb1e5e0a-f7a7-4a53-968a-3e8404fdce87/volumes" Oct 30 06:57:58.119965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.119951 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d781ce9b-5bf3-4ea9-870b-b22a77d02929" path="/var/lib/kubelet/pods/d781ce9b-5bf3-4ea9-870b-b22a77d02929/volumes" Oct 30 06:57:58.120711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.120694 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b9c3f0-9eda-481c-bf6a-86f72cb837fd" path="/var/lib/kubelet/pods/d9b9c3f0-9eda-481c-bf6a-86f72cb837fd/volumes" Oct 30 06:57:58.121052 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.121040 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4ef9dd-2519-4506-a03e-3205ad65d331" path="/var/lib/kubelet/pods/fc4ef9dd-2519-4506-a03e-3205ad65d331/volumes" Oct 30 06:57:58.202430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.202394 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd"] Oct 30 06:57:58.203042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203028 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" Oct 30 06:57:58.203089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203044 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" Oct 30 06:57:58.203089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203063 5001 cpu_manager.go:414] "containerMap: removing container" podUID="d781ce9b-5bf3-4ea9-870b-b22a77d02929" containerName="payload-processing" Oct 30 06:57:58.203089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203068 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d781ce9b-5bf3-4ea9-870b-b22a77d02929" containerName="payload-processing" Oct 30 06:57:58.203089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203075 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cb1e5e0a-f7a7-4a53-968a-3e8404fdce87" containerName="payload-pre-processing" Oct 30 06:57:58.203089 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203080 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1e5e0a-f7a7-4a53-968a-3e8404fdce87" containerName="payload-pre-processing" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203099 5001 cpu_manager.go:414] "containerMap: removing container" podUID="067fb315-b20e-42f3-b792-d081d921b529" containerName="payload-pre-processing" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203104 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067fb315-b20e-42f3-b792-d081d921b529" containerName="payload-pre-processing" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203113 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fc4ef9dd-2519-4506-a03e-3205ad65d331" containerName="payload-processing" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203117 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4ef9dd-2519-4506-a03e-3205ad65d331" containerName="payload-processing" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203132 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203136 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203146 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e2e7da2c-4f88-48b6-a09b-4c0371075b69" containerName="revoke-keys" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203150 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e7da2c-4f88-48b6-a09b-4c0371075b69" containerName="revoke-keys" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203162 5001 cpu_manager.go:414] "containerMap: removing container" podUID="635a07fa-78b2-457e-9f7e-f41d6610b469" containerName="payload-pre-processing" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203167 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635a07fa-78b2-457e-9f7e-f41d6610b469" containerName="payload-pre-processing" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203179 5001 cpu_manager.go:414] "containerMap: removing container" podUID="05977063-0120-460c-bee1-8ae684ea6ac7" containerName="payload-pre-processing" Oct 30 06:57:58.203272 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203184 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05977063-0120-460c-bee1-8ae684ea6ac7" containerName="payload-pre-processing" Oct 30 06:57:58.203701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203290 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" Oct 30 06:57:58.203701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203300 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e7da2c-4f88-48b6-a09b-4c0371075b69" containerName="revoke-keys" Oct 30 06:57:58.203701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203322 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="635a07fa-78b2-457e-9f7e-f41d6610b469" containerName="payload-pre-processing" Oct 30 06:57:58.203701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203330 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="05977063-0120-460c-bee1-8ae684ea6ac7" containerName="payload-pre-processing" Oct 30 06:57:58.203701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203338 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7e9f40-f353-436a-b771-ea6223ac02db" containerName="istio-proxy" Oct 30 06:57:58.203701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203345 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d781ce9b-5bf3-4ea9-870b-b22a77d02929" containerName="payload-processing" Oct 30 06:57:58.203701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203351 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1e5e0a-f7a7-4a53-968a-3e8404fdce87" containerName="payload-pre-processing" Oct 30 06:57:58.203701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203359 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="067fb315-b20e-42f3-b792-d081d921b529" containerName="payload-pre-processing" Oct 30 06:57:58.203701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.203365 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4ef9dd-2519-4506-a03e-3205ad65d331" containerName="payload-processing" Oct 30 06:57:58.208668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.208649 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:57:58.223245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.223216 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:57:58.226727 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.226704 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj"] Oct 30 06:57:58.231572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.231551 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd"] Oct 30 06:57:58.231668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.231652 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:57:58.232555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.232533 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:57:58.232555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.232550 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:57:58.232742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.232541 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:57:58.254446 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.254391 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj"] Oct 30 06:57:58.340752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.340676 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b51b12dd-ntxhj\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:57:58.340752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.340732 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d9390f6-cbhrd\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:57:58.340927 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.340846 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j5bp\" (UniqueName: \"kubernetes.io/projected/b0a08015-0f9b-43ac-8777-2456a1d806f0-kube-api-access-6j5bp\") pod \"maas-api-revoke-sub-b51b12dd-ntxhj\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:57:58.340987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.340969 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b51b12dd-ntxhj\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:57:58.341026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.341002 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d9390f6-cbhrd\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:57:58.341026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.341023 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mblc8\" (UniqueName: \"kubernetes.io/projected/020d4ae7-95d3-4808-ab77-27a92b540d20-kube-api-access-mblc8\") pod \"maas-api-revoke-sub-6d9390f6-cbhrd\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:57:58.442883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.442844 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d9390f6-cbhrd\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:57:58.443099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.442940 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j5bp\" (UniqueName: \"kubernetes.io/projected/b0a08015-0f9b-43ac-8777-2456a1d806f0-kube-api-access-6j5bp\") pod \"maas-api-revoke-sub-b51b12dd-ntxhj\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:57:58.443099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.443072 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b51b12dd-ntxhj\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:57:58.443234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.443109 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d9390f6-cbhrd\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:57:58.443234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.443138 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblc8\" (UniqueName: \"kubernetes.io/projected/020d4ae7-95d3-4808-ab77-27a92b540d20-kube-api-access-mblc8\") pod \"maas-api-revoke-sub-6d9390f6-cbhrd\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:57:58.443378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.443279 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b51b12dd-ntxhj\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:57:58.443748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.443718 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d9390f6-cbhrd\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:57:58.444262 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.444238 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b51b12dd-ntxhj\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:57:58.445989 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.445965 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d9390f6-cbhrd\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:57:58.446093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.446011 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b51b12dd-ntxhj\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:57:58.466013 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.465988 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblc8\" (UniqueName: \"kubernetes.io/projected/020d4ae7-95d3-4808-ab77-27a92b540d20-kube-api-access-mblc8\") pod \"maas-api-revoke-sub-6d9390f6-cbhrd\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:57:58.466139 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.466088 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j5bp\" (UniqueName: \"kubernetes.io/projected/b0a08015-0f9b-43ac-8777-2456a1d806f0-kube-api-access-6j5bp\") pod \"maas-api-revoke-sub-b51b12dd-ntxhj\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:57:58.523242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.523214 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:57:58.558587 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.558560 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:57:58.585912 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.585881 5001 generic.go:356] "Generic (PLEG): container finished" podID="067fb315-b20e-42f3-b792-d081d921b529" containerID="753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575" exitCode=0 Oct 30 06:57:58.586097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.585961 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" event={"ID":"067fb315-b20e-42f3-b792-d081d921b529","Type":"ContainerDied","Data":"753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575"} Oct 30 06:57:58.586097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.585998 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" event={"ID":"067fb315-b20e-42f3-b792-d081d921b529","Type":"ContainerDied","Data":"0787ebc307f5c0147904d48187d2d6615d0c9894bf9c6d4e792896efc3f3bda4"} Oct 30 06:57:58.586097 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.586018 5001 scope.go:122] "RemoveContainer" containerID="753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575" Oct 30 06:57:58.586283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.586100 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb" Oct 30 06:57:58.615506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.612028 5001 scope.go:122] "RemoveContainer" containerID="753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575" Oct 30 06:57:58.619853 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:57:58.615796 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575\": container with ID starting with 753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575 not found: ID does not exist" containerID="753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575" Oct 30 06:57:58.619853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.615837 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575"} err="failed to get container status \"753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575\": rpc error: code = NotFound desc = could not find container \"753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575\": container with ID starting with 753287a49d7e74364bcc08973d637ff35b7ea3174b44465899660b9134408575 not found: ID does not exist" Oct 30 06:57:58.728811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.724740 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb"] Oct 30 06:57:58.738762 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.738724 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8baf7e-9ff764875-7dprb"] Oct 30 06:57:58.844783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.844747 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd"] Oct 30 06:57:58.846005 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:57:58.845944 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020d4ae7_95d3_4808_ab77_27a92b540d20.slice/crio-c1cbf3014267452e89858337a837d363ac4b1f3df2b2b1e7c3d4f6ee5d034c3a WatchSource:0}: Error finding container c1cbf3014267452e89858337a837d363ac4b1f3df2b2b1e7c3d4f6ee5d034c3a: Status 404 returned error can't find the container with id c1cbf3014267452e89858337a837d363ac4b1f3df2b2b1e7c3d4f6ee5d034c3a Oct 30 06:57:58.905841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:58.905817 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj"] Oct 30 06:57:58.907076 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:57:58.907049 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a08015_0f9b_43ac_8777_2456a1d806f0.slice/crio-29d3d64b0a96d2df3f910c0d5c1fd7512019c6ddb9b7f544ab203b99ce6c8fe5 WatchSource:0}: Error finding container 29d3d64b0a96d2df3f910c0d5c1fd7512019c6ddb9b7f544ab203b99ce6c8fe5: Status 404 returned error can't find the container with id 29d3d64b0a96d2df3f910c0d5c1fd7512019c6ddb9b7f544ab203b99ce6c8fe5 Oct 30 06:57:59.033675 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:59.033639 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h"] Oct 30 06:57:59.037710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:59.037680 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805e52b6-dvl7h"] Oct 30 06:57:59.600351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:59.600227 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" event={"ID":"020d4ae7-95d3-4808-ab77-27a92b540d20","Type":"ContainerStarted","Data":"4b3fe56a83dd0f15eccf604eea469b49a4dac166c8160fc124c7c43fb0ed0c7a"} Oct 30 06:57:59.600351 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:59.600293 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" event={"ID":"020d4ae7-95d3-4808-ab77-27a92b540d20","Type":"ContainerStarted","Data":"c1cbf3014267452e89858337a837d363ac4b1f3df2b2b1e7c3d4f6ee5d034c3a"} Oct 30 06:57:59.602514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:59.602290 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" event={"ID":"b0a08015-0f9b-43ac-8777-2456a1d806f0","Type":"ContainerStarted","Data":"643b4ab1802f3f20c5afec5881496598b8dee51e6bf343b87e7281d302778111"} Oct 30 06:57:59.602514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:59.602370 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" event={"ID":"b0a08015-0f9b-43ac-8777-2456a1d806f0","Type":"ContainerStarted","Data":"29d3d64b0a96d2df3f910c0d5c1fd7512019c6ddb9b7f544ab203b99ce6c8fe5"} Oct 30 06:57:59.643479 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:59.643431 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" podStartSLOduration=1.643417766 podStartE2EDuration="1.643417766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:59.639186042 +0000 UTC m=+3893.635143051" watchObservedRunningTime="2026-10-30 06:57:59.643417766 +0000 UTC m=+3893.639374776" Oct 30 06:57:59.686698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:57:59.686644 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" podStartSLOduration=1.686629232 podStartE2EDuration="1.686629232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:59.682300077 +0000 UTC m=+3893.678257087" watchObservedRunningTime="2026-10-30 06:57:59.686629232 +0000 UTC m=+3893.682586241" Oct 30 06:58:00.119497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:00.119464 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="067fb315-b20e-42f3-b792-d081d921b529" path="/var/lib/kubelet/pods/067fb315-b20e-42f3-b792-d081d921b529/volumes" Oct 30 06:58:00.119935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:00.119920 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f" path="/var/lib/kubelet/pods/47ddf0da-b0bf-4c5e-9a96-7dc402a97d5f/volumes" Oct 30 06:58:00.728284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:00.726508 5001 generic.go:356] "Generic (PLEG): container finished" podID="b0a08015-0f9b-43ac-8777-2456a1d806f0" containerID="643b4ab1802f3f20c5afec5881496598b8dee51e6bf343b87e7281d302778111" exitCode=0 Oct 30 06:58:00.728284 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:00.726971 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" event={"ID":"b0a08015-0f9b-43ac-8777-2456a1d806f0","Type":"ContainerDied","Data":"643b4ab1802f3f20c5afec5881496598b8dee51e6bf343b87e7281d302778111"} Oct 30 06:58:00.735467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:00.733605 5001 generic.go:356] "Generic (PLEG): container finished" podID="020d4ae7-95d3-4808-ab77-27a92b540d20" containerID="4b3fe56a83dd0f15eccf604eea469b49a4dac166c8160fc124c7c43fb0ed0c7a" exitCode=0 Oct 30 06:58:00.735467 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:00.733772 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" event={"ID":"020d4ae7-95d3-4808-ab77-27a92b540d20","Type":"ContainerDied","Data":"4b3fe56a83dd0f15eccf604eea469b49a4dac166c8160fc124c7c43fb0ed0c7a"} Oct 30 06:58:00.739679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:00.739644 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" podUID="0711132a-6143-43f8-91a8-979b147a168d" containerName="payload-pre-processing" containerID="cri-o://f45195ccdf2322bafead410660548b17d383f43ca4b6aa79098ef51a84c09a51" gracePeriod=25 Oct 30 06:58:00.919882 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:00.919843 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr"] Oct 30 06:58:00.924960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:00.924930 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a5f459-sfkkr"] Oct 30 06:58:01.466232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.466185 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" containerID="cri-o://4d47e2e1dd35b3c43751118a4ba5511e48785c2dad5c9f797e1a0f6aa8f2b61f" gracePeriod=45 Oct 30 06:58:01.744415 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.744386 5001 generic.go:356] "Generic (PLEG): container finished" podID="6a721547-d091-4208-b0bd-4264a62c37cd" containerID="4d47e2e1dd35b3c43751118a4ba5511e48785c2dad5c9f797e1a0f6aa8f2b61f" exitCode=0 Oct 30 06:58:01.744847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.744425 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" event={"ID":"6a721547-d091-4208-b0bd-4264a62c37cd","Type":"ContainerDied","Data":"4d47e2e1dd35b3c43751118a4ba5511e48785c2dad5c9f797e1a0f6aa8f2b61f"} Oct 30 06:58:01.744847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.744487 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" event={"ID":"6a721547-d091-4208-b0bd-4264a62c37cd","Type":"ContainerDied","Data":"1072ba8450863d4f9db27c62969dfefca0a98b0b92e3e6a1063783df7ec2cfe7"} Oct 30 06:58:01.744847 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.744499 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1072ba8450863d4f9db27c62969dfefca0a98b0b92e3e6a1063783df7ec2cfe7" Oct 30 06:58:01.760435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.760399 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:58:01.926862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.926821 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-model-cache\") pod \"6a721547-d091-4208-b0bd-4264a62c37cd\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " Oct 30 06:58:01.927050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.927017 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-tmp-dir\") pod \"6a721547-d091-4208-b0bd-4264a62c37cd\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " Oct 30 06:58:01.927148 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.927075 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-home\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-home\") pod \"6a721547-d091-4208-b0bd-4264a62c37cd\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " Oct 30 06:58:01.927220 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.927172 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a721547-d091-4208-b0bd-4264a62c37cd-kube-api-access-lvtzg\" (UniqueName: \"kubernetes.io/projected/6a721547-d091-4208-b0bd-4264a62c37cd-kube-api-access-lvtzg\") pod \"6a721547-d091-4208-b0bd-4264a62c37cd\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " Oct 30 06:58:01.927281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.927240 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-dshm\") pod \"6a721547-d091-4208-b0bd-4264a62c37cd\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " Oct 30 06:58:01.927281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.927272 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a721547-d091-4208-b0bd-4264a62c37cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a721547-d091-4208-b0bd-4264a62c37cd-tls-certs\") pod \"6a721547-d091-4208-b0bd-4264a62c37cd\" (UID: \"6a721547-d091-4208-b0bd-4264a62c37cd\") " Oct 30 06:58:01.927423 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.927288 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-tmp-dir" pod "6a721547-d091-4208-b0bd-4264a62c37cd" (UID: "6a721547-d091-4208-b0bd-4264a62c37cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.927742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.927710 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-model-cache" pod "6a721547-d091-4208-b0bd-4264a62c37cd" (UID: "6a721547-d091-4208-b0bd-4264a62c37cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.928237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.928197 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.928237 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.928227 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.928833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.928799 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-home" pod "6a721547-d091-4208-b0bd-4264a62c37cd" (UID: "6a721547-d091-4208-b0bd-4264a62c37cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.937560 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.937514 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-dshm" pod "6a721547-d091-4208-b0bd-4264a62c37cd" (UID: "6a721547-d091-4208-b0bd-4264a62c37cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.938779 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.938748 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a721547-d091-4208-b0bd-4264a62c37cd-tls-certs" pod "6a721547-d091-4208-b0bd-4264a62c37cd" (UID: "6a721547-d091-4208-b0bd-4264a62c37cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:01.938914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.938881 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a721547-d091-4208-b0bd-4264a62c37cd-kube-api-access-lvtzg" pod "6a721547-d091-4208-b0bd-4264a62c37cd" (UID: "6a721547-d091-4208-b0bd-4264a62c37cd"). InnerVolumeSpecName "kube-api-access-lvtzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:01.977408 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.977380 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:58:01.985528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:01.985467 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:58:02.029430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.029396 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvtzg\" (UniqueName: \"kubernetes.io/projected/6a721547-d091-4208-b0bd-4264a62c37cd-kube-api-access-lvtzg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.029430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.029421 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.029430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.029433 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a721547-d091-4208-b0bd-4264a62c37cd-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.029430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.029441 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a721547-d091-4208-b0bd-4264a62c37cd-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.117911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.117880 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9702025a-0ba2-4515-9118-d700256f7aa8" path="/var/lib/kubelet/pods/9702025a-0ba2-4515-9118-d700256f7aa8/volumes" Oct 30 06:58:02.129924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.129896 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-xks-ca\") pod \"b0a08015-0f9b-43ac-8777-2456a1d806f0\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " Oct 30 06:58:02.130053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.129975 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a08015-0f9b-43ac-8777-2456a1d806f0-kube-api-access-6j5bp\" (UniqueName: \"kubernetes.io/projected/b0a08015-0f9b-43ac-8777-2456a1d806f0-kube-api-access-6j5bp\") pod \"b0a08015-0f9b-43ac-8777-2456a1d806f0\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " Oct 30 06:58:02.130053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.130009 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020d4ae7-95d3-4808-ab77-27a92b540d20-kube-api-access-mblc8\" (UniqueName: \"kubernetes.io/projected/020d4ae7-95d3-4808-ab77-27a92b540d20-kube-api-access-mblc8\") pod \"020d4ae7-95d3-4808-ab77-27a92b540d20\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " Oct 30 06:58:02.130157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.130147 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-openshift-ca\") pod \"020d4ae7-95d3-4808-ab77-27a92b540d20\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " Oct 30 06:58:02.130221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.130204 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-xks-ca\") pod \"020d4ae7-95d3-4808-ab77-27a92b540d20\" (UID: \"020d4ae7-95d3-4808-ab77-27a92b540d20\") " Oct 30 06:58:02.130269 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.130259 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-openshift-ca\") pod \"b0a08015-0f9b-43ac-8777-2456a1d806f0\" (UID: \"b0a08015-0f9b-43ac-8777-2456a1d806f0\") " Oct 30 06:58:02.130601 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.130555 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-openshift-ca" pod "020d4ae7-95d3-4808-ab77-27a92b540d20" (UID: "020d4ae7-95d3-4808-ab77-27a92b540d20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:02.130808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.130620 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-openshift-ca" pod "b0a08015-0f9b-43ac-8777-2456a1d806f0" (UID: "b0a08015-0f9b-43ac-8777-2456a1d806f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:02.130879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.130846 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.130879 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.130863 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.132280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.132239 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-xks-ca" pod "b0a08015-0f9b-43ac-8777-2456a1d806f0" (UID: "b0a08015-0f9b-43ac-8777-2456a1d806f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:02.132907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.132883 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020d4ae7-95d3-4808-ab77-27a92b540d20-kube-api-access-mblc8" pod "020d4ae7-95d3-4808-ab77-27a92b540d20" (UID: "020d4ae7-95d3-4808-ab77-27a92b540d20"). InnerVolumeSpecName "kube-api-access-mblc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:02.133020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.132999 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-xks-ca" pod "020d4ae7-95d3-4808-ab77-27a92b540d20" (UID: "020d4ae7-95d3-4808-ab77-27a92b540d20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:02.133083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.133052 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a08015-0f9b-43ac-8777-2456a1d806f0-kube-api-access-6j5bp" pod "b0a08015-0f9b-43ac-8777-2456a1d806f0" (UID: "b0a08015-0f9b-43ac-8777-2456a1d806f0"). InnerVolumeSpecName "kube-api-access-6j5bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:02.232844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.232814 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020d4ae7-95d3-4808-ab77-27a92b540d20-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.232844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.232843 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a08015-0f9b-43ac-8777-2456a1d806f0-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.233283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.232855 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j5bp\" (UniqueName: \"kubernetes.io/projected/b0a08015-0f9b-43ac-8777-2456a1d806f0-kube-api-access-6j5bp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.233283 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.232868 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mblc8\" (UniqueName: \"kubernetes.io/projected/020d4ae7-95d3-4808-ab77-27a92b540d20-kube-api-access-mblc8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.776663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.776568 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" Oct 30 06:58:02.776663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.776589 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51b12dd-ntxhj" event={"ID":"b0a08015-0f9b-43ac-8777-2456a1d806f0","Type":"ContainerDied","Data":"29d3d64b0a96d2df3f910c0d5c1fd7512019c6ddb9b7f544ab203b99ce6c8fe5"} Oct 30 06:58:02.776663 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.776661 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29d3d64b0a96d2df3f910c0d5c1fd7512019c6ddb9b7f544ab203b99ce6c8fe5" Oct 30 06:58:02.778495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.778473 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" Oct 30 06:58:02.778651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.778502 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9390f6-cbhrd" event={"ID":"020d4ae7-95d3-4808-ab77-27a92b540d20","Type":"ContainerDied","Data":"c1cbf3014267452e89858337a837d363ac4b1f3df2b2b1e7c3d4f6ee5d034c3a"} Oct 30 06:58:02.778651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.778549 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1cbf3014267452e89858337a837d363ac4b1f3df2b2b1e7c3d4f6ee5d034c3a" Oct 30 06:58:02.778770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.778699 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr" Oct 30 06:58:02.899593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.899563 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr"] Oct 30 06:58:02.910274 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:02.910235 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/premium-w0-a519e9-kserve-69798b5c88-5lwzr"] Oct 30 06:58:03.607584 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:03.607547 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" podUID="0711132a-6143-43f8-91a8-979b147a168d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:58:04.118795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:04.118763 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" path="/var/lib/kubelet/pods/6a721547-d091-4208-b0bd-4264a62c37cd/volumes" Oct 30 06:58:04.539572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:04.539485 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6"] Oct 30 06:58:04.539797 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:04.539760 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" containerID="cri-o://8637ee5f1fca0d2f4c2c8ff8d465834513b894929972632909ad99899eccc68d" gracePeriod=30 Oct 30 06:58:05.169769 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.169734 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4"] Oct 30 06:58:05.170414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.170399 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" Oct 30 06:58:05.170484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.170416 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" Oct 30 06:58:05.170484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.170426 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b0a08015-0f9b-43ac-8777-2456a1d806f0" containerName="revoke-keys" Oct 30 06:58:05.170484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.170431 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a08015-0f9b-43ac-8777-2456a1d806f0" containerName="revoke-keys" Oct 30 06:58:05.170484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.170469 5001 cpu_manager.go:414] "containerMap: removing container" podUID="020d4ae7-95d3-4808-ab77-27a92b540d20" containerName="revoke-keys" Oct 30 06:58:05.170484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.170474 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020d4ae7-95d3-4808-ab77-27a92b540d20" containerName="revoke-keys" Oct 30 06:58:05.170649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.170590 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="020d4ae7-95d3-4808-ab77-27a92b540d20" containerName="revoke-keys" Oct 30 06:58:05.170649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.170602 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a721547-d091-4208-b0bd-4264a62c37cd" containerName="main" Oct 30 06:58:05.170649 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.170608 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a08015-0f9b-43ac-8777-2456a1d806f0" containerName="revoke-keys" Oct 30 06:58:05.175748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.175730 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:05.194282 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.194257 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4"] Oct 30 06:58:05.200374 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.200351 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:58:05.200581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.200387 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:58:05.211360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.211332 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:58:05.211548 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.211416 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:58:05.289888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.289852 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:05.290095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.289976 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch4dv\" (UniqueName: \"kubernetes.io/projected/8c61a33a-c956-4ce5-a213-74795a9b1a30-kube-api-access-ch4dv\") pod \"maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:05.290095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.290081 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:05.391900 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.391866 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:05.392104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.391973 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch4dv\" (UniqueName: \"kubernetes.io/projected/8c61a33a-c956-4ce5-a213-74795a9b1a30-kube-api-access-ch4dv\") pod \"maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:05.392104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.392041 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:05.392804 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.392782 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:05.394668 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.394639 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:05.414944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.414919 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch4dv\" (UniqueName: \"kubernetes.io/projected/8c61a33a-c956-4ce5-a213-74795a9b1a30-kube-api-access-ch4dv\") pod \"maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:05.491738 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.491647 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:05.813702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:05.813611 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4"] Oct 30 06:58:05.826609 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:58:05.826516 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c61a33a_c956_4ce5_a213_74795a9b1a30.slice/crio-c6437c8cadbf1ca83cfa5a1de44b1f734968a81f2bbf9b6a9172cc3c2b1b7d28 WatchSource:0}: Error finding container c6437c8cadbf1ca83cfa5a1de44b1f734968a81f2bbf9b6a9172cc3c2b1b7d28: Status 404 returned error can't find the container with id c6437c8cadbf1ca83cfa5a1de44b1f734968a81f2bbf9b6a9172cc3c2b1b7d28 Oct 30 06:58:06.828918 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:06.828883 5001 generic.go:356] "Generic (PLEG): container finished" podID="8c61a33a-c956-4ce5-a213-74795a9b1a30" containerID="63ba0a99904515568d439f2f181599f97af3ac833a7073c24e81ecbb3faa9c4e" exitCode=0 Oct 30 06:58:06.829389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:06.828983 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" event={"ID":"8c61a33a-c956-4ce5-a213-74795a9b1a30","Type":"ContainerDied","Data":"63ba0a99904515568d439f2f181599f97af3ac833a7073c24e81ecbb3faa9c4e"} Oct 30 06:58:06.829389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:06.829042 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" event={"ID":"8c61a33a-c956-4ce5-a213-74795a9b1a30","Type":"ContainerStarted","Data":"c6437c8cadbf1ca83cfa5a1de44b1f734968a81f2bbf9b6a9172cc3c2b1b7d28"} Oct 30 06:58:07.208898 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:07.208868 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:58:07.221494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:07.221466 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 06:58:08.020164 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.020132 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:08.128714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.128684 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-openshift-ca\") pod \"8c61a33a-c956-4ce5-a213-74795a9b1a30\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " Oct 30 06:58:08.128881 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.128723 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-xks-ca\") pod \"8c61a33a-c956-4ce5-a213-74795a9b1a30\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " Oct 30 06:58:08.128971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.128909 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c61a33a-c956-4ce5-a213-74795a9b1a30-kube-api-access-ch4dv\" (UniqueName: \"kubernetes.io/projected/8c61a33a-c956-4ce5-a213-74795a9b1a30-kube-api-access-ch4dv\") pod \"8c61a33a-c956-4ce5-a213-74795a9b1a30\" (UID: \"8c61a33a-c956-4ce5-a213-74795a9b1a30\") " Oct 30 06:58:08.129082 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.129054 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-openshift-ca" pod "8c61a33a-c956-4ce5-a213-74795a9b1a30" (UID: "8c61a33a-c956-4ce5-a213-74795a9b1a30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:08.129556 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.129480 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:08.131119 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.131092 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c61a33a-c956-4ce5-a213-74795a9b1a30-kube-api-access-ch4dv" pod "8c61a33a-c956-4ce5-a213-74795a9b1a30" (UID: "8c61a33a-c956-4ce5-a213-74795a9b1a30"). InnerVolumeSpecName "kube-api-access-ch4dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:08.131213 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.131138 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-xks-ca" pod "8c61a33a-c956-4ce5-a213-74795a9b1a30" (UID: "8c61a33a-c956-4ce5-a213-74795a9b1a30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:08.231568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.231527 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c61a33a-c956-4ce5-a213-74795a9b1a30-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:08.231568 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.231558 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch4dv\" (UniqueName: \"kubernetes.io/projected/8c61a33a-c956-4ce5-a213-74795a9b1a30-kube-api-access-ch4dv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:08.271607 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:58:08.271570 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:58:08.466136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.466041 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd"] Oct 30 06:58:08.466770 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.466295 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" containerID="cri-o://82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315" gracePeriod=30 Oct 30 06:58:08.849767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.849687 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" Oct 30 06:58:08.849911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.849684 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4" event={"ID":"8c61a33a-c956-4ce5-a213-74795a9b1a30","Type":"ContainerDied","Data":"c6437c8cadbf1ca83cfa5a1de44b1f734968a81f2bbf9b6a9172cc3c2b1b7d28"} Oct 30 06:58:08.849911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:08.849833 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6437c8cadbf1ca83cfa5a1de44b1f734968a81f2bbf9b6a9172cc3c2b1b7d28" Oct 30 06:58:09.075405 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.075371 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt"] Oct 30 06:58:09.076452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.076431 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8c61a33a-c956-4ce5-a213-74795a9b1a30" containerName="revoke-keys" Oct 30 06:58:09.076515 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.076454 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c61a33a-c956-4ce5-a213-74795a9b1a30" containerName="revoke-keys" Oct 30 06:58:09.076678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.076667 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c61a33a-c956-4ce5-a213-74795a9b1a30" containerName="revoke-keys" Oct 30 06:58:09.081076 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.081056 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:09.097385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.097299 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:58:09.098008 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.097987 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:58:09.098096 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.098081 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:58:09.098170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.098109 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt"] Oct 30 06:58:09.104185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.104123 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:58:09.247165 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.247130 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:09.247355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.247286 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:09.247473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.247456 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dl2q\" (UniqueName: \"kubernetes.io/projected/4fd4b166-cc66-43bf-bbd2-4b03994537c2-kube-api-access-8dl2q\") pod \"maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:09.349011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.348977 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:09.349212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.349079 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:09.349212 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.349166 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dl2q\" (UniqueName: \"kubernetes.io/projected/4fd4b166-cc66-43bf-bbd2-4b03994537c2-kube-api-access-8dl2q\") pod \"maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:09.349711 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.349690 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:09.351638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.351609 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:09.372380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.372354 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dl2q\" (UniqueName: \"kubernetes.io/projected/4fd4b166-cc66-43bf-bbd2-4b03994537c2-kube-api-access-8dl2q\") pod \"maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:09.398084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.398049 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:09.710090 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.710063 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt"] Oct 30 06:58:09.711858 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:58:09.711830 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd4b166_cc66_43bf_bbd2_4b03994537c2.slice/crio-763022210c85427be179315247918783fa1897b57d04891794e7221f25ef6555 WatchSource:0}: Error finding container 763022210c85427be179315247918783fa1897b57d04891794e7221f25ef6555: Status 404 returned error can't find the container with id 763022210c85427be179315247918783fa1897b57d04891794e7221f25ef6555 Oct 30 06:58:09.864050 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.864012 5001 generic.go:356] "Generic (PLEG): container finished" podID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerID="8637ee5f1fca0d2f4c2c8ff8d465834513b894929972632909ad99899eccc68d" exitCode=0 Oct 30 06:58:09.864225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.864084 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" event={"ID":"65dfe1f4-528d-4b3d-9bf4-7a83d57130be","Type":"ContainerDied","Data":"8637ee5f1fca0d2f4c2c8ff8d465834513b894929972632909ad99899eccc68d"} Oct 30 06:58:09.864225 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.864156 5001 scope.go:122] "RemoveContainer" containerID="d04ed4c49dbc4f48669f06a06f13cb4c0e43f762aba221f9af6c3aa693d2e433" Oct 30 06:58:09.865880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:09.865856 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" event={"ID":"4fd4b166-cc66-43bf-bbd2-4b03994537c2","Type":"ContainerStarted","Data":"763022210c85427be179315247918783fa1897b57d04891794e7221f25ef6555"} Oct 30 06:58:10.245801 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.245768 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:58:10.362919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.362885 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-credential-socket\") pod \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " Oct 30 06:58:10.363198 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.363171 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istiod-ca-cert\") pod \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " Oct 30 06:58:10.363411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.363275 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-data\") pod \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " Oct 30 06:58:10.363411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.363339 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-certs\") pod \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " Oct 30 06:58:10.363411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.363367 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-envoy\") pod \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " Oct 30 06:58:10.363411 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.363409 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-kube-api-access-sf7wx\" (UniqueName: \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-kube-api-access-sf7wx\") pod \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " Oct 30 06:58:10.363691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.363463 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-socket\") pod \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " Oct 30 06:58:10.363691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.363526 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-token\" (UniqueName: \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-token\") pod \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " Oct 30 06:58:10.363691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.363617 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-podinfo\") pod \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\" (UID: \"65dfe1f4-528d-4b3d-9bf4-7a83d57130be\") " Oct 30 06:58:10.363866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.363176 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-credential-socket" pod "65dfe1f4-528d-4b3d-9bf4-7a83d57130be" (UID: "65dfe1f4-528d-4b3d-9bf4-7a83d57130be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:10.363866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.363538 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istiod-ca-cert" pod "65dfe1f4-528d-4b3d-9bf4-7a83d57130be" (UID: "65dfe1f4-528d-4b3d-9bf4-7a83d57130be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:10.363866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.363770 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-certs" pod "65dfe1f4-528d-4b3d-9bf4-7a83d57130be" (UID: "65dfe1f4-528d-4b3d-9bf4-7a83d57130be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:10.364271 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.364248 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-data" pod "65dfe1f4-528d-4b3d-9bf4-7a83d57130be" (UID: "65dfe1f4-528d-4b3d-9bf4-7a83d57130be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:10.364785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.364757 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-socket" pod "65dfe1f4-528d-4b3d-9bf4-7a83d57130be" (UID: "65dfe1f4-528d-4b3d-9bf4-7a83d57130be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:10.364864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.364809 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.364864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.364829 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.364864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.364844 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.364864 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.364857 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.367053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.367026 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-envoy" pod "65dfe1f4-528d-4b3d-9bf4-7a83d57130be" (UID: "65dfe1f4-528d-4b3d-9bf4-7a83d57130be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:10.367215 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.367195 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-kube-api-access-sf7wx" pod "65dfe1f4-528d-4b3d-9bf4-7a83d57130be" (UID: "65dfe1f4-528d-4b3d-9bf4-7a83d57130be"). InnerVolumeSpecName "kube-api-access-sf7wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:10.367364 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.367290 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-token" pod "65dfe1f4-528d-4b3d-9bf4-7a83d57130be" (UID: "65dfe1f4-528d-4b3d-9bf4-7a83d57130be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:10.368780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.368756 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-podinfo" pod "65dfe1f4-528d-4b3d-9bf4-7a83d57130be" (UID: "65dfe1f4-528d-4b3d-9bf4-7a83d57130be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:58:10.466495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.466460 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.466495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.466497 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.466706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.466511 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf7wx\" (UniqueName: \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-kube-api-access-sf7wx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.466706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.466525 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.466706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.466537 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65dfe1f4-528d-4b3d-9bf4-7a83d57130be-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.880053 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.880019 5001 generic.go:356] "Generic (PLEG): container finished" podID="4fd4b166-cc66-43bf-bbd2-4b03994537c2" containerID="5a21cdd36cd20c5f564dd1781af55e0a099f2e2c1802899505d8c3de6e004fe2" exitCode=0 Oct 30 06:58:10.880280 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.880122 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" event={"ID":"4fd4b166-cc66-43bf-bbd2-4b03994537c2","Type":"ContainerDied","Data":"5a21cdd36cd20c5f564dd1781af55e0a099f2e2c1802899505d8c3de6e004fe2"} Oct 30 06:58:10.882114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.882076 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" event={"ID":"65dfe1f4-528d-4b3d-9bf4-7a83d57130be","Type":"ContainerDied","Data":"1413520afd8f37354950a4bc86eed9d9e785ef85adc3c0574ede712cfabf0cb0"} Oct 30 06:58:10.882240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.882137 5001 scope.go:122] "RemoveContainer" containerID="8637ee5f1fca0d2f4c2c8ff8d465834513b894929972632909ad99899eccc68d" Oct 30 06:58:10.882240 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.882176 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6" Oct 30 06:58:10.958825 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.958795 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6"] Oct 30 06:58:10.965780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:10.965749 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8baf7e-openshift-default-78bb4856df-z8sn6"] Oct 30 06:58:12.039348 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.037047 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn"] Oct 30 06:58:12.055268 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.055232 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9da4cdf2-mqlkn"] Oct 30 06:58:12.086626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.086582 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:12.117713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.117679 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bec1809-a925-451b-9d14-7bdf5377bfb3" path="/var/lib/kubelet/pods/2bec1809-a925-451b-9d14-7bdf5377bfb3/volumes" Oct 30 06:58:12.118187 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.118161 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" path="/var/lib/kubelet/pods/65dfe1f4-528d-4b3d-9bf4-7a83d57130be/volumes" Oct 30 06:58:12.192049 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.191953 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd4b166-cc66-43bf-bbd2-4b03994537c2-kube-api-access-8dl2q\" (UniqueName: \"kubernetes.io/projected/4fd4b166-cc66-43bf-bbd2-4b03994537c2-kube-api-access-8dl2q\") pod \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " Oct 30 06:58:12.192250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.192076 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-xks-ca\") pod \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " Oct 30 06:58:12.192250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.192178 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-openshift-ca\") pod \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\" (UID: \"4fd4b166-cc66-43bf-bbd2-4b03994537c2\") " Oct 30 06:58:12.193421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.193391 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-openshift-ca" pod "4fd4b166-cc66-43bf-bbd2-4b03994537c2" (UID: "4fd4b166-cc66-43bf-bbd2-4b03994537c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:12.196124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.196079 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-xks-ca" pod "4fd4b166-cc66-43bf-bbd2-4b03994537c2" (UID: "4fd4b166-cc66-43bf-bbd2-4b03994537c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:12.196442 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.196418 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd4b166-cc66-43bf-bbd2-4b03994537c2-kube-api-access-8dl2q" pod "4fd4b166-cc66-43bf-bbd2-4b03994537c2" (UID: "4fd4b166-cc66-43bf-bbd2-4b03994537c2"). InnerVolumeSpecName "kube-api-access-8dl2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:12.294279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.294023 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:12.294279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.294059 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd4b166-cc66-43bf-bbd2-4b03994537c2-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:12.294279 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.294074 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dl2q\" (UniqueName: \"kubernetes.io/projected/4fd4b166-cc66-43bf-bbd2-4b03994537c2-kube-api-access-8dl2q\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:12.304796 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.304767 5001 scope.go:122] "RemoveContainer" containerID="4d47e2e1dd35b3c43751118a4ba5511e48785c2dad5c9f797e1a0f6aa8f2b61f" Oct 30 06:58:12.317460 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.317441 5001 scope.go:122] "RemoveContainer" containerID="a7c024c27300d139e27a78d177a720daa5267c76858453b4712a99e1fbf64ca2" Oct 30 06:58:12.327853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.327828 5001 scope.go:122] "RemoveContainer" containerID="102fef86ee7a0a308e7978dbc2de26e64539b0e9cd5318575dbefa75b70d60d0" Oct 30 06:58:12.338180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.338147 5001 scope.go:122] "RemoveContainer" containerID="fc00c777fdd4c8fa0449765de51f013a18136d3b4560ac043e5cdc2c333a31fa" Oct 30 06:58:12.905704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.905677 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" Oct 30 06:58:12.905888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.905703 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt" event={"ID":"4fd4b166-cc66-43bf-bbd2-4b03994537c2","Type":"ContainerDied","Data":"763022210c85427be179315247918783fa1897b57d04891794e7221f25ef6555"} Oct 30 06:58:12.905888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:12.905751 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="763022210c85427be179315247918783fa1897b57d04891794e7221f25ef6555" Oct 30 06:58:13.610330 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.609767 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" podUID="0711132a-6143-43f8-91a8-979b147a168d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:58:13.634403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.634367 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:58:13.799441 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.799414 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:58:13.915555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.915500 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-istio-token\" (UniqueName: \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-istio-token\") pod \"313ac877-7f04-420f-a158-32c97424341a\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " Oct 30 06:58:13.915754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.915633 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-kube-api-access-plfm6\" (UniqueName: \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-kube-api-access-plfm6\") pod \"313ac877-7f04-420f-a158-32c97424341a\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " Oct 30 06:58:13.915754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.915661 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-data\") pod \"313ac877-7f04-420f-a158-32c97424341a\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " Oct 30 06:58:13.915754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.915733 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313ac877-7f04-420f-a158-32c97424341a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313ac877-7f04-420f-a158-32c97424341a-istiod-ca-cert\") pod \"313ac877-7f04-420f-a158-32c97424341a\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " Oct 30 06:58:13.915965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.915785 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-envoy\") pod \"313ac877-7f04-420f-a158-32c97424341a\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " Oct 30 06:58:13.915965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.915838 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-credential-socket\") pod \"313ac877-7f04-420f-a158-32c97424341a\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " Oct 30 06:58:13.916174 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.916150 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-credential-socket" pod "313ac877-7f04-420f-a158-32c97424341a" (UID: "313ac877-7f04-420f-a158-32c97424341a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:13.916260 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.916166 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-data" pod "313ac877-7f04-420f-a158-32c97424341a" (UID: "313ac877-7f04-420f-a158-32c97424341a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:13.917379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.917020 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-certs\") pod \"313ac877-7f04-420f-a158-32c97424341a\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " Oct 30 06:58:13.917379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.917085 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/313ac877-7f04-420f-a158-32c97424341a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313ac877-7f04-420f-a158-32c97424341a-istio-podinfo\") pod \"313ac877-7f04-420f-a158-32c97424341a\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " Oct 30 06:58:13.917379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.917152 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-socket\") pod \"313ac877-7f04-420f-a158-32c97424341a\" (UID: \"313ac877-7f04-420f-a158-32c97424341a\") " Oct 30 06:58:13.918221 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.918194 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:13.918391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.918223 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:13.918391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.918226 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-kube-api-access-plfm6" pod "313ac877-7f04-420f-a158-32c97424341a" (UID: "313ac877-7f04-420f-a158-32c97424341a"). InnerVolumeSpecName "kube-api-access-plfm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:13.918391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.918269 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-envoy" pod "313ac877-7f04-420f-a158-32c97424341a" (UID: "313ac877-7f04-420f-a158-32c97424341a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:13.918391 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.918346 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-istio-token" pod "313ac877-7f04-420f-a158-32c97424341a" (UID: "313ac877-7f04-420f-a158-32c97424341a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:13.918808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.918782 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-certs" pod "313ac877-7f04-420f-a158-32c97424341a" (UID: "313ac877-7f04-420f-a158-32c97424341a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:13.918808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.918789 5001 generic.go:356] "Generic (PLEG): container finished" podID="313ac877-7f04-420f-a158-32c97424341a" containerID="82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315" exitCode=0 Oct 30 06:58:13.918969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.918836 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" event={"ID":"313ac877-7f04-420f-a158-32c97424341a","Type":"ContainerDied","Data":"82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315"} Oct 30 06:58:13.918969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.918902 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" event={"ID":"313ac877-7f04-420f-a158-32c97424341a","Type":"ContainerDied","Data":"0884a02c0c57a8b43ea08c03958d771899e2a67f889c5b38b616dd8b3603a35f"} Oct 30 06:58:13.918969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.918914 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-socket" pod "313ac877-7f04-420f-a158-32c97424341a" (UID: "313ac877-7f04-420f-a158-32c97424341a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:13.918969 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.918931 5001 scope.go:122] "RemoveContainer" containerID="82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315" Oct 30 06:58:13.919223 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.919012 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd" Oct 30 06:58:13.919651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.919627 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313ac877-7f04-420f-a158-32c97424341a-istiod-ca-cert" pod "313ac877-7f04-420f-a158-32c97424341a" (UID: "313ac877-7f04-420f-a158-32c97424341a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:13.921604 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.921570 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/313ac877-7f04-420f-a158-32c97424341a-istio-podinfo" pod "313ac877-7f04-420f-a158-32c97424341a" (UID: "313ac877-7f04-420f-a158-32c97424341a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:58:13.952003 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.951975 5001 scope.go:122] "RemoveContainer" containerID="569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1" Oct 30 06:58:13.968131 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.968106 5001 scope.go:122] "RemoveContainer" containerID="82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315" Oct 30 06:58:13.968537 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:58:13.968504 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315\": container with ID starting with 82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315 not found: ID does not exist" containerID="82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315" Oct 30 06:58:13.968626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.968552 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315"} err="failed to get container status \"82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315\": rpc error: code = NotFound desc = could not find container \"82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315\": container with ID starting with 82cc45ddbff21926b4ccb900431888519fa3dca93d3e2c8538503f353bff4315 not found: ID does not exist" Oct 30 06:58:13.968626 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.968580 5001 scope.go:122] "RemoveContainer" containerID="569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1" Oct 30 06:58:13.968924 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:58:13.968902 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1\": container with ID starting with 569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1 not found: ID does not exist" containerID="569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1" Oct 30 06:58:13.969015 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:13.968937 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1"} err="failed to get container status \"569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1\": rpc error: code = NotFound desc = could not find container \"569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1\": container with ID starting with 569f4b98c60b8c40f19fe5775a0fcfe7829dcb7f642ca58f61b116c4f62d75b1 not found: ID does not exist" Oct 30 06:58:14.020480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:14.020358 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plfm6\" (UniqueName: \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-kube-api-access-plfm6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:14.020480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:14.020400 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313ac877-7f04-420f-a158-32c97424341a-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:14.020480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:14.020417 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:14.020480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:14.020430 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:14.020480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:14.020443 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313ac877-7f04-420f-a158-32c97424341a-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:14.020480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:14.020455 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313ac877-7f04-420f-a158-32c97424341a-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:14.020480 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:14.020468 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313ac877-7f04-420f-a158-32c97424341a-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:14.262277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:14.262236 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd"] Oct 30 06:58:14.269457 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:14.269427 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a5f459-openshift-default-66968fd697-b22xd"] Oct 30 06:58:15.667357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:15.666029 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr"] Oct 30 06:58:16.121959 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:16.121922 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313ac877-7f04-420f-a158-32c97424341a" path="/var/lib/kubelet/pods/313ac877-7f04-420f-a158-32c97424341a/volumes" Oct 30 06:58:18.186492 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:18.186390 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf"] Oct 30 06:58:18.187841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:18.187813 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-89f101-dn2lf"] Oct 30 06:58:20.160207 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:20.159148 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e7da2c-4f88-48b6-a09b-4c0371075b69" path="/var/lib/kubelet/pods/e2e7da2c-4f88-48b6-a09b-4c0371075b69/volumes" Oct 30 06:58:20.668196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:20.668148 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" podUID="f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5" containerName="payload-pre-processing" containerID="cri-o://f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b" gracePeriod=25 Oct 30 06:58:20.996938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:20.996904 5001 generic.go:356] "Generic (PLEG): container finished" podID="0711132a-6143-43f8-91a8-979b147a168d" containerID="f45195ccdf2322bafead410660548b17d383f43ca4b6aa79098ef51a84c09a51" exitCode=0 Oct 30 06:58:20.997079 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:20.996981 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" event={"ID":"0711132a-6143-43f8-91a8-979b147a168d","Type":"ContainerDied","Data":"f45195ccdf2322bafead410660548b17d383f43ca4b6aa79098ef51a84c09a51"} Oct 30 06:58:21.021503 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:21.021476 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:58:21.029345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:21.029319 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0711132a-6143-43f8-91a8-979b147a168d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0711132a-6143-43f8-91a8-979b147a168d-plugins-config-volume\") pod \"0711132a-6143-43f8-91a8-979b147a168d\" (UID: \"0711132a-6143-43f8-91a8-979b147a168d\") " Oct 30 06:58:21.029518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:21.029369 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0711132a-6143-43f8-91a8-979b147a168d-kube-api-access-g7r9z\" (UniqueName: \"kubernetes.io/projected/0711132a-6143-43f8-91a8-979b147a168d-kube-api-access-g7r9z\") pod \"0711132a-6143-43f8-91a8-979b147a168d\" (UID: \"0711132a-6143-43f8-91a8-979b147a168d\") " Oct 30 06:58:21.029640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:21.029608 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0711132a-6143-43f8-91a8-979b147a168d-plugins-config-volume" pod "0711132a-6143-43f8-91a8-979b147a168d" (UID: "0711132a-6143-43f8-91a8-979b147a168d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:21.029982 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:21.029963 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0711132a-6143-43f8-91a8-979b147a168d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:21.031680 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:21.031659 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0711132a-6143-43f8-91a8-979b147a168d-kube-api-access-g7r9z" pod "0711132a-6143-43f8-91a8-979b147a168d" (UID: "0711132a-6143-43f8-91a8-979b147a168d"). InnerVolumeSpecName "kube-api-access-g7r9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:21.130917 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:21.130884 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7r9z\" (UniqueName: \"kubernetes.io/projected/0711132a-6143-43f8-91a8-979b147a168d-kube-api-access-g7r9z\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:22.008168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:22.008137 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" Oct 30 06:58:22.008168 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:22.008155 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv" event={"ID":"0711132a-6143-43f8-91a8-979b147a168d","Type":"ContainerDied","Data":"0dafeb23fe5946bb3c9e92b0314491111cb1a133801f0891061442ed5245b48d"} Oct 30 06:58:22.008752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:22.008226 5001 scope.go:122] "RemoveContainer" containerID="f45195ccdf2322bafead410660548b17d383f43ca4b6aa79098ef51a84c09a51" Oct 30 06:58:22.060960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:22.060927 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv"] Oct 30 06:58:22.071833 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:22.071809 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89f101-d7dd45c55-mkdnv"] Oct 30 06:58:22.118250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:22.118221 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0711132a-6143-43f8-91a8-979b147a168d" path="/var/lib/kubelet/pods/0711132a-6143-43f8-91a8-979b147a168d/volumes" Oct 30 06:58:23.069414 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:23.069373 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" podUID="f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:58:26.532744 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:26.532705 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z"] Oct 30 06:58:26.533194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:26.532983 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" containerID="cri-o://0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9" gracePeriod=30 Oct 30 06:58:27.177740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.177694 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff"] Oct 30 06:58:27.178730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.178706 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0711132a-6143-43f8-91a8-979b147a168d" containerName="payload-pre-processing" Oct 30 06:58:27.178730 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.178731 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0711132a-6143-43f8-91a8-979b147a168d" containerName="payload-pre-processing" Oct 30 06:58:27.179009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.178746 5001 cpu_manager.go:414] "containerMap: removing container" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" Oct 30 06:58:27.179009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.178754 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" Oct 30 06:58:27.179009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.178778 5001 cpu_manager.go:414] "containerMap: removing container" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" Oct 30 06:58:27.179009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.178785 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" Oct 30 06:58:27.179009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.178809 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4fd4b166-cc66-43bf-bbd2-4b03994537c2" containerName="revoke-keys" Oct 30 06:58:27.179009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.178818 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd4b166-cc66-43bf-bbd2-4b03994537c2" containerName="revoke-keys" Oct 30 06:58:27.179009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.178870 5001 cpu_manager.go:414] "containerMap: removing container" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" Oct 30 06:58:27.179009 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.178879 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" Oct 30 06:58:27.179462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.179035 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0711132a-6143-43f8-91a8-979b147a168d" containerName="payload-pre-processing" Oct 30 06:58:27.179462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.179049 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" Oct 30 06:58:27.179462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.179060 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" Oct 30 06:58:27.179462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.179070 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd4b166-cc66-43bf-bbd2-4b03994537c2" containerName="revoke-keys" Oct 30 06:58:27.179462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.179082 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" Oct 30 06:58:27.179462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.179091 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="313ac877-7f04-420f-a158-32c97424341a" containerName="istio-proxy" Oct 30 06:58:27.184891 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.184867 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:27.200827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.200798 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff"] Oct 30 06:58:27.213844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.206786 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:58:27.213844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.206878 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:58:27.213844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.206919 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:58:27.213844 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.206884 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:58:27.216246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.216218 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:27.216514 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.216495 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:27.216704 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.216689 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8pn\" (UniqueName: \"kubernetes.io/projected/fe71be15-c165-4be8-a7b7-4fa19dc39f97-kube-api-access-2h8pn\") pod \"maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:27.321317 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.321265 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:27.321526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.321381 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:27.321526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.321463 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8pn\" (UniqueName: \"kubernetes.io/projected/fe71be15-c165-4be8-a7b7-4fa19dc39f97-kube-api-access-2h8pn\") pod \"maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:27.322424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.322399 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:27.324051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.324023 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:27.346128 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.346099 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8pn\" (UniqueName: \"kubernetes.io/projected/fe71be15-c165-4be8-a7b7-4fa19dc39f97-kube-api-access-2h8pn\") pod \"maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:27.501916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.501835 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:27.836007 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:27.835966 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff"] Oct 30 06:58:27.837581 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:58:27.837547 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe71be15_c165_4be8_a7b7_4fa19dc39f97.slice/crio-68204512df0e10efa725564ee12899d6ea39637fdbd8ea9d7781793d545356ce WatchSource:0}: Error finding container 68204512df0e10efa725564ee12899d6ea39637fdbd8ea9d7781793d545356ce: Status 404 returned error can't find the container with id 68204512df0e10efa725564ee12899d6ea39637fdbd8ea9d7781793d545356ce Oct 30 06:58:28.067700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:28.067664 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" event={"ID":"fe71be15-c165-4be8-a7b7-4fa19dc39f97","Type":"ContainerStarted","Data":"68204512df0e10efa725564ee12899d6ea39637fdbd8ea9d7781793d545356ce"} Oct 30 06:58:29.078820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:29.078780 5001 generic.go:356] "Generic (PLEG): container finished" podID="fe71be15-c165-4be8-a7b7-4fa19dc39f97" containerID="03bd6fda191942837076aede3a630e3ceda00914f5a4f975264c3757a5454a33" exitCode=0 Oct 30 06:58:29.079229 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:29.078859 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" event={"ID":"fe71be15-c165-4be8-a7b7-4fa19dc39f97","Type":"ContainerDied","Data":"03bd6fda191942837076aede3a630e3ceda00914f5a4f975264c3757a5454a33"} Oct 30 06:58:30.260129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:30.260102 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:30.361084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:30.361042 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-openshift-ca\") pod \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " Oct 30 06:58:30.361295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:30.361100 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-xks-ca\") pod \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " Oct 30 06:58:30.361295 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:30.361194 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe71be15-c165-4be8-a7b7-4fa19dc39f97-kube-api-access-2h8pn\" (UniqueName: \"kubernetes.io/projected/fe71be15-c165-4be8-a7b7-4fa19dc39f97-kube-api-access-2h8pn\") pod \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\" (UID: \"fe71be15-c165-4be8-a7b7-4fa19dc39f97\") " Oct 30 06:58:30.361464 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:30.361448 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-openshift-ca" pod "fe71be15-c165-4be8-a7b7-4fa19dc39f97" (UID: "fe71be15-c165-4be8-a7b7-4fa19dc39f97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:30.361703 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:30.361681 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:30.363463 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:30.363435 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe71be15-c165-4be8-a7b7-4fa19dc39f97-kube-api-access-2h8pn" pod "fe71be15-c165-4be8-a7b7-4fa19dc39f97" (UID: "fe71be15-c165-4be8-a7b7-4fa19dc39f97"). InnerVolumeSpecName "kube-api-access-2h8pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:30.363590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:30.363480 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-xks-ca" pod "fe71be15-c165-4be8-a7b7-4fa19dc39f97" (UID: "fe71be15-c165-4be8-a7b7-4fa19dc39f97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:30.463261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:30.463219 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h8pn\" (UniqueName: \"kubernetes.io/projected/fe71be15-c165-4be8-a7b7-4fa19dc39f97-kube-api-access-2h8pn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:30.463261 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:30.463250 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71be15-c165-4be8-a7b7-4fa19dc39f97-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.099253 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:31.099223 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" Oct 30 06:58:31.099465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:31.099250 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff" event={"ID":"fe71be15-c165-4be8-a7b7-4fa19dc39f97","Type":"ContainerDied","Data":"68204512df0e10efa725564ee12899d6ea39637fdbd8ea9d7781793d545356ce"} Oct 30 06:58:31.099465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:31.099365 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68204512df0e10efa725564ee12899d6ea39637fdbd8ea9d7781793d545356ce" Oct 30 06:58:31.349087 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:31.349045 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8"] Oct 30 06:58:31.370360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:31.370299 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a519e9-rvjt8"] Oct 30 06:58:31.871732 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:31.871697 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt"] Oct 30 06:58:31.883688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:31.883660 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ad1661-rjkgt"] Oct 30 06:58:31.898058 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:31.898005 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:58:32.010777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.010733 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-envoy\") pod \"29524a8b-436b-4b95-a88c-8b4d433474b1\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " Oct 30 06:58:32.028410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.023778 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-envoy" pod "29524a8b-436b-4b95-a88c-8b4d433474b1" (UID: "29524a8b-436b-4b95-a88c-8b4d433474b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:32.028651 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.028510 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-kube-api-access-twfhm\" (UniqueName: \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-kube-api-access-twfhm\") pod \"29524a8b-436b-4b95-a88c-8b4d433474b1\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " Oct 30 06:58:32.028807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.028786 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-socket\") pod \"29524a8b-436b-4b95-a88c-8b4d433474b1\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " Oct 30 06:58:32.029117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.029048 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-podinfo\") pod \"29524a8b-436b-4b95-a88c-8b4d433474b1\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " Oct 30 06:58:32.029232 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.029165 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-token\" (UniqueName: \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-token\") pod \"29524a8b-436b-4b95-a88c-8b4d433474b1\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " Oct 30 06:58:32.029322 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.029268 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29524a8b-436b-4b95-a88c-8b4d433474b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29524a8b-436b-4b95-a88c-8b4d433474b1-istiod-ca-cert\") pod \"29524a8b-436b-4b95-a88c-8b4d433474b1\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " Oct 30 06:58:32.029389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.029344 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-certs\") pod \"29524a8b-436b-4b95-a88c-8b4d433474b1\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " Oct 30 06:58:32.029453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.029437 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-credential-socket\") pod \"29524a8b-436b-4b95-a88c-8b4d433474b1\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " Oct 30 06:58:32.029516 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.029474 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-data\") pod \"29524a8b-436b-4b95-a88c-8b4d433474b1\" (UID: \"29524a8b-436b-4b95-a88c-8b4d433474b1\") " Oct 30 06:58:32.029643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.029619 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-socket" pod "29524a8b-436b-4b95-a88c-8b4d433474b1" (UID: "29524a8b-436b-4b95-a88c-8b4d433474b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:32.035576 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.030289 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-data" pod "29524a8b-436b-4b95-a88c-8b4d433474b1" (UID: "29524a8b-436b-4b95-a88c-8b4d433474b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:32.035733 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.030914 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29524a8b-436b-4b95-a88c-8b4d433474b1-istiod-ca-cert" pod "29524a8b-436b-4b95-a88c-8b4d433474b1" (UID: "29524a8b-436b-4b95-a88c-8b4d433474b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:32.035863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.030979 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-certs" pod "29524a8b-436b-4b95-a88c-8b4d433474b1" (UID: "29524a8b-436b-4b95-a88c-8b4d433474b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:32.035996 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.031176 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-credential-socket" pod "29524a8b-436b-4b95-a88c-8b4d433474b1" (UID: "29524a8b-436b-4b95-a88c-8b4d433474b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:32.036114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.030076 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:32.052393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.043297 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-kube-api-access-twfhm" pod "29524a8b-436b-4b95-a88c-8b4d433474b1" (UID: "29524a8b-436b-4b95-a88c-8b4d433474b1"). InnerVolumeSpecName "kube-api-access-twfhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:32.052393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.043493 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-token" pod "29524a8b-436b-4b95-a88c-8b4d433474b1" (UID: "29524a8b-436b-4b95-a88c-8b4d433474b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:32.052393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.046976 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-podinfo" pod "29524a8b-436b-4b95-a88c-8b4d433474b1" (UID: "29524a8b-436b-4b95-a88c-8b4d433474b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:58:32.111046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.111014 5001 generic.go:356] "Generic (PLEG): container finished" podID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerID="0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9" exitCode=0 Oct 30 06:58:32.111231 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.111131 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" Oct 30 06:58:32.117750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.117723 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd4b166-cc66-43bf-bbd2-4b03994537c2" path="/var/lib/kubelet/pods/4fd4b166-cc66-43bf-bbd2-4b03994537c2/volumes" Oct 30 06:58:32.118197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.118182 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b5f1a3-b8b4-4188-aef9-991a892e6995" path="/var/lib/kubelet/pods/a9b5f1a3-b8b4-4188-aef9-991a892e6995/volumes" Oct 30 06:58:32.118786 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.118760 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" event={"ID":"29524a8b-436b-4b95-a88c-8b4d433474b1","Type":"ContainerDied","Data":"0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9"} Oct 30 06:58:32.118904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.118800 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z" event={"ID":"29524a8b-436b-4b95-a88c-8b4d433474b1","Type":"ContainerDied","Data":"ede6fde7f0d2184ca064ed963610d012e849dd1abcc363da69b7108b09161fe4"} Oct 30 06:58:32.118904 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.118832 5001 scope.go:122] "RemoveContainer" containerID="0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9" Oct 30 06:58:32.132938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.132868 5001 scope.go:122] "RemoveContainer" containerID="3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9" Oct 30 06:58:32.137180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.137154 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:32.137276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.137186 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:32.137276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.137200 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twfhm\" (UniqueName: \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-kube-api-access-twfhm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:32.137276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.137214 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:32.137276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.137228 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:32.137276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.137240 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29524a8b-436b-4b95-a88c-8b4d433474b1-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:32.137276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.137252 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29524a8b-436b-4b95-a88c-8b4d433474b1-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:32.137276 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.137263 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29524a8b-436b-4b95-a88c-8b4d433474b1-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:32.146059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.146036 5001 scope.go:122] "RemoveContainer" containerID="0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9" Oct 30 06:58:32.146355 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:58:32.146333 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9\": container with ID starting with 0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9 not found: ID does not exist" containerID="0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9" Oct 30 06:58:32.146418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.146365 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9"} err="failed to get container status \"0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9\": rpc error: code = NotFound desc = could not find container \"0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9\": container with ID starting with 0a557dd5c3676137267d8f4598d6f48417b8fe3fdcb12e69ea9c68c6b4f6c1f9 not found: ID does not exist" Oct 30 06:58:32.146418 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.146381 5001 scope.go:122] "RemoveContainer" containerID="3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9" Oct 30 06:58:32.146641 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:58:32.146623 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9\": container with ID starting with 3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9 not found: ID does not exist" containerID="3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9" Oct 30 06:58:32.146685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.146648 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9"} err="failed to get container status \"3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9\": rpc error: code = NotFound desc = could not find container \"3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9\": container with ID starting with 3ff4c701b4eae7b4c76b10fbba9b10f3a5fe6cc688a588920d8ef2c4b1ab95a9 not found: ID does not exist" Oct 30 06:58:32.164854 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.164793 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z"] Oct 30 06:58:32.169462 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.169441 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-89f101-openshift-default-5fdc59467f-75g5z"] Oct 30 06:58:32.389635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.389589 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4"] Oct 30 06:58:32.395793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:32.395761 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-be48b4-q9kf4"] Oct 30 06:58:33.069042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:33.069002 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" podUID="f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:58:34.117873 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:34.117842 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" path="/var/lib/kubelet/pods/29524a8b-436b-4b95-a88c-8b4d433474b1/volumes" Oct 30 06:58:34.118346 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:34.118329 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c61a33a-c956-4ce5-a213-74795a9b1a30" path="/var/lib/kubelet/pods/8c61a33a-c956-4ce5-a213-74795a9b1a30/volumes" Oct 30 06:58:37.653687 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:37.653649 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b"] Oct 30 06:58:37.688497 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:37.688463 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v"] Oct 30 06:58:38.274652 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:58:38.274613 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:58:40.608852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.608790 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p"] Oct 30 06:58:40.635461 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.635416 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" containerID="cri-o://57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9" gracePeriod=30 Oct 30 06:58:40.952104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.952067 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9"] Oct 30 06:58:40.953054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.953028 5001 cpu_manager.go:414] "containerMap: removing container" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" Oct 30 06:58:40.953054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.953053 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dfe1f4-528d-4b3d-9bf4-7a83d57130be" containerName="istio-proxy" Oct 30 06:58:40.953258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.953131 5001 cpu_manager.go:414] "containerMap: removing container" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" Oct 30 06:58:40.953258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.953139 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" Oct 30 06:58:40.953258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.953158 5001 cpu_manager.go:414] "containerMap: removing container" podUID="fe71be15-c165-4be8-a7b7-4fa19dc39f97" containerName="revoke-keys" Oct 30 06:58:40.953258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.953165 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe71be15-c165-4be8-a7b7-4fa19dc39f97" containerName="revoke-keys" Oct 30 06:58:40.953258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.953177 5001 cpu_manager.go:414] "containerMap: removing container" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" Oct 30 06:58:40.953258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.953184 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" Oct 30 06:58:40.953664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.953341 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" Oct 30 06:58:40.953664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.953358 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe71be15-c165-4be8-a7b7-4fa19dc39f97" containerName="revoke-keys" Oct 30 06:58:40.953664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.953370 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="29524a8b-436b-4b95-a88c-8b4d433474b1" containerName="istio-proxy" Oct 30 06:58:40.958170 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.958148 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:40.961577 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.961549 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4v5v\" (UniqueName: \"kubernetes.io/projected/dc83cc44-158d-4461-913b-9070e289ae17-kube-api-access-s4v5v\") pod \"maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:40.961719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.961672 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:40.961787 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.961755 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:40.979518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.972752 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:58:40.979518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.972831 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:58:40.980477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:40.980444 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9"] Oct 30 06:58:41.024889 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.024853 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:58:41.062385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.062351 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-kube-api-access-4nlrp\" (UniqueName: \"kubernetes.io/projected/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-kube-api-access-4nlrp\") pod \"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5\" (UID: \"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5\") " Oct 30 06:58:41.062608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.062430 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-plugins-config-volume\") pod \"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5\" (UID: \"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5\") " Oct 30 06:58:41.062608 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.062552 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:41.062764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.062653 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4v5v\" (UniqueName: \"kubernetes.io/projected/dc83cc44-158d-4461-913b-9070e289ae17-kube-api-access-s4v5v\") pod \"maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:41.062764 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.062723 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:41.062875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.062843 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-plugins-config-volume" pod "f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5" (UID: "f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:41.063379 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.063357 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:41.065402 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.065367 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-kube-api-access-4nlrp" pod "f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5" (UID: "f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5"). InnerVolumeSpecName "kube-api-access-4nlrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:41.067217 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.067194 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:58:41.067393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.067377 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:58:41.081324 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.081278 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:41.088529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.088501 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4v5v\" (UniqueName: \"kubernetes.io/projected/dc83cc44-158d-4461-913b-9070e289ae17-kube-api-access-s4v5v\") pod \"maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:41.163862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.163781 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nlrp\" (UniqueName: \"kubernetes.io/projected/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-kube-api-access-4nlrp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:41.163862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.163807 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:41.212073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.212043 5001 generic.go:356] "Generic (PLEG): container finished" podID="f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5" containerID="f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b" exitCode=0 Oct 30 06:58:41.212255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.212092 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" Oct 30 06:58:41.212255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.212115 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" event={"ID":"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5","Type":"ContainerDied","Data":"f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b"} Oct 30 06:58:41.212255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.212176 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr" event={"ID":"f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5","Type":"ContainerDied","Data":"ff6e20ef36caf08ee9a5d8dff5e25aec2196e068ea85129c0cb78810ef7a19d6"} Oct 30 06:58:41.212255 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.212199 5001 scope.go:122] "RemoveContainer" containerID="f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b" Oct 30 06:58:41.224586 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.224567 5001 scope.go:122] "RemoveContainer" containerID="f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b" Oct 30 06:58:41.224882 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:58:41.224863 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b\": container with ID starting with f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b not found: ID does not exist" containerID="f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b" Oct 30 06:58:41.224939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.224890 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b"} err="failed to get container status \"f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b\": rpc error: code = NotFound desc = could not find container \"f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b\": container with ID starting with f350fd8b32b1e47b6c4b126f6c3296d73c1a3e0e33cbf0f772118439298b8d3b not found: ID does not exist" Oct 30 06:58:41.263710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.263680 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr"] Oct 30 06:58:41.270194 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.270171 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-be48b4-fdf6dd687-hfqfr"] Oct 30 06:58:41.289505 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.289482 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm"] Oct 30 06:58:41.290159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.290140 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5" containerName="payload-pre-processing" Oct 30 06:58:41.290159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.290159 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5" containerName="payload-pre-processing" Oct 30 06:58:41.290407 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.290391 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5" containerName="payload-pre-processing" Oct 30 06:58:41.295388 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.295370 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:41.306512 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.306490 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:41.314973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.314950 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm"] Oct 30 06:58:41.366399 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.366362 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5p8\" (UniqueName: \"kubernetes.io/projected/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-kube-api-access-sq5p8\") pod \"maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:41.366539 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.366506 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:41.366599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.366586 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:41.467621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.467528 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5p8\" (UniqueName: \"kubernetes.io/projected/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-kube-api-access-sq5p8\") pod \"maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:41.467621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.467616 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:41.467874 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.467690 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:41.468294 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.468272 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:41.470297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.470274 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:41.490957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.490928 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5p8\" (UniqueName: \"kubernetes.io/projected/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-kube-api-access-sq5p8\") pod \"maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:41.610265 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.610233 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:41.625205 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.625181 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9"] Oct 30 06:58:41.627169 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:58:41.627146 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc83cc44_158d_4461_913b_9070e289ae17.slice/crio-d89b304382ee4b8168b88fe19f1fa8fb713407e2b1e6786595ff62c2212ad00b WatchSource:0}: Error finding container d89b304382ee4b8168b88fe19f1fa8fb713407e2b1e6786595ff62c2212ad00b: Status 404 returned error can't find the container with id d89b304382ee4b8168b88fe19f1fa8fb713407e2b1e6786595ff62c2212ad00b Oct 30 06:58:41.924791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:41.924764 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm"] Oct 30 06:58:41.926622 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:58:41.926598 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48de4f37_3d8f_46e4_aae4_cadfb388c5c5.slice/crio-2a30d3ec50ce4d0eba8554a56970d7d800be0db93bc4d016e176abe08b2da1a5 WatchSource:0}: Error finding container 2a30d3ec50ce4d0eba8554a56970d7d800be0db93bc4d016e176abe08b2da1a5: Status 404 returned error can't find the container with id 2a30d3ec50ce4d0eba8554a56970d7d800be0db93bc4d016e176abe08b2da1a5 Oct 30 06:58:42.118916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:42.118884 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5" path="/var/lib/kubelet/pods/f30a5651-954e-4a58-b0a0-c6e7ba9d1ae5/volumes" Oct 30 06:58:42.222581 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:42.222493 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" event={"ID":"48de4f37-3d8f-46e4-aae4-cadfb388c5c5","Type":"ContainerStarted","Data":"2a30d3ec50ce4d0eba8554a56970d7d800be0db93bc4d016e176abe08b2da1a5"} Oct 30 06:58:42.224185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:42.224158 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" event={"ID":"dc83cc44-158d-4461-913b-9070e289ae17","Type":"ContainerStarted","Data":"92d6b6bd9d5588c251317e37e9de333087eb2f6b7763776af0bbd8ffd981e2ee"} Oct 30 06:58:42.224377 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:42.224209 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" event={"ID":"dc83cc44-158d-4461-913b-9070e289ae17","Type":"ContainerStarted","Data":"d89b304382ee4b8168b88fe19f1fa8fb713407e2b1e6786595ff62c2212ad00b"} Oct 30 06:58:42.307950 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:42.267465 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" podStartSLOduration=2.267451361 podStartE2EDuration="2.267451361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:58:42.263288161 +0000 UTC m=+3936.259245183" watchObservedRunningTime="2026-10-30 06:58:42.267451361 +0000 UTC m=+3936.263408371" Oct 30 06:58:42.654688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:42.654650 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" podUID="63f1a036-b3e2-44e7-a082-862bf2070478" containerName="payload-processing" containerID="cri-o://8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab" gracePeriod=25 Oct 30 06:58:42.689713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:42.689673 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" podUID="b65ca078-eaa5-4bf4-a8c1-0856876434f1" containerName="payload-pre-processing" containerID="cri-o://7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204" gracePeriod=25 Oct 30 06:58:43.235451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:43.235417 5001 generic.go:356] "Generic (PLEG): container finished" podID="48de4f37-3d8f-46e4-aae4-cadfb388c5c5" containerID="5e1174c0f56bfdb6968e17addc0c00202cb3c392ba8e83843d63d890e9f0fb8f" exitCode=0 Oct 30 06:58:43.235652 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:43.235498 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" event={"ID":"48de4f37-3d8f-46e4-aae4-cadfb388c5c5","Type":"ContainerDied","Data":"5e1174c0f56bfdb6968e17addc0c00202cb3c392ba8e83843d63d890e9f0fb8f"} Oct 30 06:58:43.237297 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:43.237269 5001 generic.go:356] "Generic (PLEG): container finished" podID="dc83cc44-158d-4461-913b-9070e289ae17" containerID="92d6b6bd9d5588c251317e37e9de333087eb2f6b7763776af0bbd8ffd981e2ee" exitCode=0 Oct 30 06:58:43.237465 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:43.237336 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" event={"ID":"dc83cc44-158d-4461-913b-9070e289ae17","Type":"ContainerDied","Data":"92d6b6bd9d5588c251317e37e9de333087eb2f6b7763776af0bbd8ffd981e2ee"} Oct 30 06:58:44.445805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.445780 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:44.448808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.448785 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:44.519245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.519165 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-openshift-ca\") pod \"dc83cc44-158d-4461-913b-9070e289ae17\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " Oct 30 06:58:44.519245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.519208 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc83cc44-158d-4461-913b-9070e289ae17-kube-api-access-s4v5v\" (UniqueName: \"kubernetes.io/projected/dc83cc44-158d-4461-913b-9070e289ae17-kube-api-access-s4v5v\") pod \"dc83cc44-158d-4461-913b-9070e289ae17\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " Oct 30 06:58:44.519528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.519362 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-openshift-ca\") pod \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " Oct 30 06:58:44.519528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.519417 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-kube-api-access-sq5p8\" (UniqueName: \"kubernetes.io/projected/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-kube-api-access-sq5p8\") pod \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " Oct 30 06:58:44.519528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.519526 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-xks-ca\") pod \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\" (UID: \"48de4f37-3d8f-46e4-aae4-cadfb388c5c5\") " Oct 30 06:58:44.519717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.519566 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-openshift-ca" pod "dc83cc44-158d-4461-913b-9070e289ae17" (UID: "dc83cc44-158d-4461-913b-9070e289ae17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:44.519717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.519633 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-xks-ca\") pod \"dc83cc44-158d-4461-913b-9070e289ae17\" (UID: \"dc83cc44-158d-4461-913b-9070e289ae17\") " Oct 30 06:58:44.519937 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.519897 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-openshift-ca" pod "48de4f37-3d8f-46e4-aae4-cadfb388c5c5" (UID: "48de4f37-3d8f-46e4-aae4-cadfb388c5c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:44.520190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.520167 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:44.520190 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.520192 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:44.522771 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.522744 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc83cc44-158d-4461-913b-9070e289ae17-kube-api-access-s4v5v" pod "dc83cc44-158d-4461-913b-9070e289ae17" (UID: "dc83cc44-158d-4461-913b-9070e289ae17"). InnerVolumeSpecName "kube-api-access-s4v5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:44.522888 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.522833 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-kube-api-access-sq5p8" pod "48de4f37-3d8f-46e4-aae4-cadfb388c5c5" (UID: "48de4f37-3d8f-46e4-aae4-cadfb388c5c5"). InnerVolumeSpecName "kube-api-access-sq5p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:44.522960 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.522894 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-xks-ca" pod "48de4f37-3d8f-46e4-aae4-cadfb388c5c5" (UID: "48de4f37-3d8f-46e4-aae4-cadfb388c5c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:44.537709 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.537684 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-xks-ca" pod "dc83cc44-158d-4461-913b-9070e289ae17" (UID: "dc83cc44-158d-4461-913b-9070e289ae17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:44.621482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.621452 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4v5v\" (UniqueName: \"kubernetes.io/projected/dc83cc44-158d-4461-913b-9070e289ae17-kube-api-access-s4v5v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:44.621482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.621480 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq5p8\" (UniqueName: \"kubernetes.io/projected/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-kube-api-access-sq5p8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:44.621482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.621491 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48de4f37-3d8f-46e4-aae4-cadfb388c5c5-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:44.621793 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:44.621500 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc83cc44-158d-4461-913b-9070e289ae17-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.261433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:45.261403 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" Oct 30 06:58:45.261635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:45.261400 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm" event={"ID":"48de4f37-3d8f-46e4-aae4-cadfb388c5c5","Type":"ContainerDied","Data":"2a30d3ec50ce4d0eba8554a56970d7d800be0db93bc4d016e176abe08b2da1a5"} Oct 30 06:58:45.261635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:45.261528 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a30d3ec50ce4d0eba8554a56970d7d800be0db93bc4d016e176abe08b2da1a5" Oct 30 06:58:45.262972 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:45.262952 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" Oct 30 06:58:45.263112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:45.263001 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9" event={"ID":"dc83cc44-158d-4461-913b-9070e289ae17","Type":"ContainerDied","Data":"d89b304382ee4b8168b88fe19f1fa8fb713407e2b1e6786595ff62c2212ad00b"} Oct 30 06:58:45.263112 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:45.263043 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d89b304382ee4b8168b88fe19f1fa8fb713407e2b1e6786595ff62c2212ad00b" Oct 30 06:58:45.946939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:45.946910 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:58:46.054197 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.054141 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-podinfo\") pod \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " Oct 30 06:58:46.054628 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.054606 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-data\") pod \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " Oct 30 06:58:46.054909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.054874 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-certs\") pod \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " Oct 30 06:58:46.055157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.055140 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-envoy\") pod \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " Oct 30 06:58:46.055354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.055338 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-token\" (UniqueName: \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-token\") pod \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " Oct 30 06:58:46.055555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.055539 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-credential-socket\") pod \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " Oct 30 06:58:46.055795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.055778 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-kube-api-access-rpdrw\" (UniqueName: \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-kube-api-access-rpdrw\") pod \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " Oct 30 06:58:46.055987 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.055972 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-socket\") pod \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " Oct 30 06:58:46.056132 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.056118 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istiod-ca-cert\") pod \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\" (UID: \"cd760fe2-e8c7-42f9-8a3f-40ec810159b4\") " Oct 30 06:58:46.057103 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.054904 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-data" pod "cd760fe2-e8c7-42f9-8a3f-40ec810159b4" (UID: "cd760fe2-e8c7-42f9-8a3f-40ec810159b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:46.057259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.055075 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-certs" pod "cd760fe2-e8c7-42f9-8a3f-40ec810159b4" (UID: "cd760fe2-e8c7-42f9-8a3f-40ec810159b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:46.057386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.057078 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istiod-ca-cert" pod "cd760fe2-e8c7-42f9-8a3f-40ec810159b4" (UID: "cd760fe2-e8c7-42f9-8a3f-40ec810159b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:46.057737 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.057717 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-credential-socket" pod "cd760fe2-e8c7-42f9-8a3f-40ec810159b4" (UID: "cd760fe2-e8c7-42f9-8a3f-40ec810159b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:46.059869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.059825 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-socket" pod "cd760fe2-e8c7-42f9-8a3f-40ec810159b4" (UID: "cd760fe2-e8c7-42f9-8a3f-40ec810159b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:46.075820 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.075766 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-podinfo" pod "cd760fe2-e8c7-42f9-8a3f-40ec810159b4" (UID: "cd760fe2-e8c7-42f9-8a3f-40ec810159b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:58:46.076011 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.075992 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-envoy" pod "cd760fe2-e8c7-42f9-8a3f-40ec810159b4" (UID: "cd760fe2-e8c7-42f9-8a3f-40ec810159b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:46.076204 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.076184 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-token" pod "cd760fe2-e8c7-42f9-8a3f-40ec810159b4" (UID: "cd760fe2-e8c7-42f9-8a3f-40ec810159b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:46.078201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.078173 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-kube-api-access-rpdrw" pod "cd760fe2-e8c7-42f9-8a3f-40ec810159b4" (UID: "cd760fe2-e8c7-42f9-8a3f-40ec810159b4"). InnerVolumeSpecName "kube-api-access-rpdrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:46.159061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.159026 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpdrw\" (UniqueName: \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-kube-api-access-rpdrw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.159062 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159061 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.159078 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.159091 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.159102 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.159112 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.159145 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.159163 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159403 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.159172 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd760fe2-e8c7-42f9-8a3f-40ec810159b4-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.276840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.276757 5001 generic.go:356] "Generic (PLEG): container finished" podID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerID="57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9" exitCode=0 Oct 30 06:58:46.276840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.276813 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" Oct 30 06:58:46.277094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.276841 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" event={"ID":"cd760fe2-e8c7-42f9-8a3f-40ec810159b4","Type":"ContainerDied","Data":"57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9"} Oct 30 06:58:46.277094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.276902 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p" event={"ID":"cd760fe2-e8c7-42f9-8a3f-40ec810159b4","Type":"ContainerDied","Data":"cc112da958e4a220e7451554181e1fc55010f3832a8e8dac59b278bbb4158730"} Oct 30 06:58:46.277094 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.276931 5001 scope.go:122] "RemoveContainer" containerID="57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9" Oct 30 06:58:46.290077 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.290047 5001 scope.go:122] "RemoveContainer" containerID="58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160" Oct 30 06:58:46.303413 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.303394 5001 scope.go:122] "RemoveContainer" containerID="57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9" Oct 30 06:58:46.303729 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:58:46.303692 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9\": container with ID starting with 57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9 not found: ID does not exist" containerID="57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9" Oct 30 06:58:46.303822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.303730 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9"} err="failed to get container status \"57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9\": rpc error: code = NotFound desc = could not find container \"57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9\": container with ID starting with 57b267834c54487a7ff7c9316f9c49e4b8d96916f6c6e394d346556f2a6f22d9 not found: ID does not exist" Oct 30 06:58:46.303822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.303753 5001 scope.go:122] "RemoveContainer" containerID="58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160" Oct 30 06:58:46.304075 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:58:46.304050 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160\": container with ID starting with 58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160 not found: ID does not exist" containerID="58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160" Oct 30 06:58:46.304124 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.304082 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160"} err="failed to get container status \"58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160\": rpc error: code = NotFound desc = could not find container \"58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160\": container with ID starting with 58ac82f16358dc316f2fe891067f4fbf8b325ce2329bcd6007d65f8c8ffe7160 not found: ID does not exist" Oct 30 06:58:46.331176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.331145 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p"] Oct 30 06:58:46.342596 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:46.342572 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-be48b4-openshift-default-5879f5498c-56q9p"] Oct 30 06:58:48.119877 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:48.119846 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" path="/var/lib/kubelet/pods/cd760fe2-e8c7-42f9-8a3f-40ec810159b4/volumes" Oct 30 06:58:50.960443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:50.960400 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" podUID="b65ca078-eaa5-4bf4-a8c1-0856876434f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:58:50.960942 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:50.960488 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" podUID="63f1a036-b3e2-44e7-a082-862bf2070478" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:58:51.752047 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:51.752008 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w"] Oct 30 06:58:52.489046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:52.488878 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4"] Oct 30 06:58:52.508161 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:52.508131 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x"] Oct 30 06:58:53.118178 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:53.118143 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff"] Oct 30 06:58:53.121999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:53.121974 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b8ca49-l9kff"] Oct 30 06:58:53.857349 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:53.857268 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:58:54.118575 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:54.118496 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe71be15-c165-4be8-a7b7-4fa19dc39f97" path="/var/lib/kubelet/pods/fe71be15-c165-4be8-a7b7-4fa19dc39f97/volumes" Oct 30 06:58:55.321638 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:55.321604 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm"] Oct 30 06:58:55.355104 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:55.355064 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f0145f-wbwjm"] Oct 30 06:58:56.118573 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:56.118536 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48de4f37-3d8f-46e4-aae4-cadfb388c5c5" path="/var/lib/kubelet/pods/48de4f37-3d8f-46e4-aae4-cadfb388c5c5/volumes" Oct 30 06:58:56.752862 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:56.752816 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" podUID="813c0acd-5bc2-42ee-9eac-133d057df75f" containerName="payload-processing" containerID="cri-o://88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388" gracePeriod=25 Oct 30 06:58:57.489955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:57.489913 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" podUID="c553d6a6-04ad-43a5-abbd-73889c0797bb" containerName="payload-processing" containerID="cri-o://01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e" gracePeriod=25 Oct 30 06:58:57.509180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:58:57.509148 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" podUID="e15ede9e-3c58-48c1-a663-1299268d6838" containerName="payload-pre-processing" containerID="cri-o://db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164" gracePeriod=25 Oct 30 06:59:00.947719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:00.947677 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" podUID="63f1a036-b3e2-44e7-a082-862bf2070478" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:00.948192 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:00.947781 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" podUID="b65ca078-eaa5-4bf4-a8c1-0856876434f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:01.260752 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.260668 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk"] Oct 30 06:59:01.261646 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261627 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" Oct 30 06:59:01.261710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261651 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" Oct 30 06:59:01.261710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261670 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" Oct 30 06:59:01.261710 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261678 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" Oct 30 06:59:01.261816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261727 5001 cpu_manager.go:414] "containerMap: removing container" podUID="48de4f37-3d8f-46e4-aae4-cadfb388c5c5" containerName="revoke-keys" Oct 30 06:59:01.261816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261734 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48de4f37-3d8f-46e4-aae4-cadfb388c5c5" containerName="revoke-keys" Oct 30 06:59:01.261816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261752 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dc83cc44-158d-4461-913b-9070e289ae17" containerName="revoke-keys" Oct 30 06:59:01.261816 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261759 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc83cc44-158d-4461-913b-9070e289ae17" containerName="revoke-keys" Oct 30 06:59:01.261958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261917 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" Oct 30 06:59:01.261958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261928 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" Oct 30 06:59:01.261958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261945 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="48de4f37-3d8f-46e4-aae4-cadfb388c5c5" containerName="revoke-keys" Oct 30 06:59:01.261958 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.261955 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc83cc44-158d-4461-913b-9070e289ae17" containerName="revoke-keys" Oct 30 06:59:01.267993 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.267968 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:01.282121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.282097 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:59:01.282852 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.282819 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk"] Oct 30 06:59:01.288991 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.288969 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:59:01.298182 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.298158 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:59:01.298299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.298283 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:59:01.317851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.317827 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:01.317990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.317971 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsrgz\" (UniqueName: \"kubernetes.io/projected/4d515235-aded-4643-a35e-e9c09fe86663-kube-api-access-gsrgz\") pod \"maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:01.318098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.318083 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:01.419361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.419326 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsrgz\" (UniqueName: \"kubernetes.io/projected/4d515235-aded-4643-a35e-e9c09fe86663-kube-api-access-gsrgz\") pod \"maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:01.419551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.419446 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:01.419551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.419504 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:01.420228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.420204 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:01.422018 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.421988 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:01.442202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.442167 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsrgz\" (UniqueName: \"kubernetes.io/projected/4d515235-aded-4643-a35e-e9c09fe86663-kube-api-access-gsrgz\") pod \"maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:01.585258 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.585166 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:01.900098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:01.900069 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk"] Oct 30 06:59:01.903059 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:59:01.903020 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d515235_aded_4643_a35e_e9c09fe86663.slice/crio-91255ff5ebba0518e6c6492ad983a192170b48f8b7cffbf9c7b1e9a87a9f34fd WatchSource:0}: Error finding container 91255ff5ebba0518e6c6492ad983a192170b48f8b7cffbf9c7b1e9a87a9f34fd: Status 404 returned error can't find the container with id 91255ff5ebba0518e6c6492ad983a192170b48f8b7cffbf9c7b1e9a87a9f34fd Oct 30 06:59:02.435800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:02.435711 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" event={"ID":"4d515235-aded-4643-a35e-e9c09fe86663","Type":"ContainerStarted","Data":"f41cfcfef8e653be0ab559f66e80314899c3327460162b94d42d74a70b925397"} Oct 30 06:59:02.435800 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:02.435766 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" event={"ID":"4d515235-aded-4643-a35e-e9c09fe86663","Type":"ContainerStarted","Data":"91255ff5ebba0518e6c6492ad983a192170b48f8b7cffbf9c7b1e9a87a9f34fd"} Oct 30 06:59:02.476202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:02.476133 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" podStartSLOduration=1.476112624 podStartE2EDuration="1.476112624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:59:02.474512955 +0000 UTC m=+3956.470469968" watchObservedRunningTime="2026-10-30 06:59:02.476112624 +0000 UTC m=+3956.472069635" Oct 30 06:59:02.972245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:02.972165 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:59:03.011014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.010987 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:59:03.053938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.053844 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65ca078-eaa5-4bf4-a8c1-0856876434f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65ca078-eaa5-4bf4-a8c1-0856876434f1-plugins-config-volume\") pod \"b65ca078-eaa5-4bf4-a8c1-0856876434f1\" (UID: \"b65ca078-eaa5-4bf4-a8c1-0856876434f1\") " Oct 30 06:59:03.053938 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.053922 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f1a036-b3e2-44e7-a082-862bf2070478-kube-api-access-jkd52\" (UniqueName: \"kubernetes.io/projected/63f1a036-b3e2-44e7-a082-862bf2070478-kube-api-access-jkd52\") pod \"63f1a036-b3e2-44e7-a082-862bf2070478\" (UID: \"63f1a036-b3e2-44e7-a082-862bf2070478\") " Oct 30 06:59:03.054211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.054072 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f1a036-b3e2-44e7-a082-862bf2070478-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f1a036-b3e2-44e7-a082-862bf2070478-plugins-config-volume\") pod \"63f1a036-b3e2-44e7-a082-862bf2070478\" (UID: \"63f1a036-b3e2-44e7-a082-862bf2070478\") " Oct 30 06:59:03.054211 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.054152 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65ca078-eaa5-4bf4-a8c1-0856876434f1-kube-api-access-twk9g\" (UniqueName: \"kubernetes.io/projected/b65ca078-eaa5-4bf4-a8c1-0856876434f1-kube-api-access-twk9g\") pod \"b65ca078-eaa5-4bf4-a8c1-0856876434f1\" (UID: \"b65ca078-eaa5-4bf4-a8c1-0856876434f1\") " Oct 30 06:59:03.057438 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.057381 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f1a036-b3e2-44e7-a082-862bf2070478-plugins-config-volume" pod "63f1a036-b3e2-44e7-a082-862bf2070478" (UID: "63f1a036-b3e2-44e7-a082-862bf2070478"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:03.058250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.058027 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65ca078-eaa5-4bf4-a8c1-0856876434f1-plugins-config-volume" pod "b65ca078-eaa5-4bf4-a8c1-0856876434f1" (UID: "b65ca078-eaa5-4bf4-a8c1-0856876434f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:03.058250 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.058042 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65ca078-eaa5-4bf4-a8c1-0856876434f1-kube-api-access-twk9g" pod "b65ca078-eaa5-4bf4-a8c1-0856876434f1" (UID: "b65ca078-eaa5-4bf4-a8c1-0856876434f1"). InnerVolumeSpecName "kube-api-access-twk9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:03.060300 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.060270 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f1a036-b3e2-44e7-a082-862bf2070478-kube-api-access-jkd52" pod "63f1a036-b3e2-44e7-a082-862bf2070478" (UID: "63f1a036-b3e2-44e7-a082-862bf2070478"). InnerVolumeSpecName "kube-api-access-jkd52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:03.155600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.155565 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f1a036-b3e2-44e7-a082-862bf2070478-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:03.155600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.155596 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twk9g\" (UniqueName: \"kubernetes.io/projected/b65ca078-eaa5-4bf4-a8c1-0856876434f1-kube-api-access-twk9g\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:03.155600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.155609 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65ca078-eaa5-4bf4-a8c1-0856876434f1-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:03.155916 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.155619 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkd52\" (UniqueName: \"kubernetes.io/projected/63f1a036-b3e2-44e7-a082-862bf2070478-kube-api-access-jkd52\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:03.446043 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.446010 5001 generic.go:356] "Generic (PLEG): container finished" podID="4d515235-aded-4643-a35e-e9c09fe86663" containerID="f41cfcfef8e653be0ab559f66e80314899c3327460162b94d42d74a70b925397" exitCode=0 Oct 30 06:59:03.446473 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.446095 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" event={"ID":"4d515235-aded-4643-a35e-e9c09fe86663","Type":"ContainerDied","Data":"f41cfcfef8e653be0ab559f66e80314899c3327460162b94d42d74a70b925397"} Oct 30 06:59:03.447590 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.447562 5001 generic.go:356] "Generic (PLEG): container finished" podID="63f1a036-b3e2-44e7-a082-862bf2070478" containerID="8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab" exitCode=0 Oct 30 06:59:03.447721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.447602 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" event={"ID":"63f1a036-b3e2-44e7-a082-862bf2070478","Type":"ContainerDied","Data":"8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab"} Oct 30 06:59:03.447721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.447607 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" Oct 30 06:59:03.447721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.447649 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b" event={"ID":"63f1a036-b3e2-44e7-a082-862bf2070478","Type":"ContainerDied","Data":"e912d4654de52a5d656e98855527e66d43f4a35a098f2850c39a656e202d6874"} Oct 30 06:59:03.447721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.447672 5001 scope.go:122] "RemoveContainer" containerID="8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab" Oct 30 06:59:03.449296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.449206 5001 generic.go:356] "Generic (PLEG): container finished" podID="b65ca078-eaa5-4bf4-a8c1-0856876434f1" containerID="7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204" exitCode=0 Oct 30 06:59:03.449296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.449246 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" Oct 30 06:59:03.449296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.449333 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" event={"ID":"b65ca078-eaa5-4bf4-a8c1-0856876434f1","Type":"ContainerDied","Data":"7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204"} Oct 30 06:59:03.449296 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.449384 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v" event={"ID":"b65ca078-eaa5-4bf4-a8c1-0856876434f1","Type":"ContainerDied","Data":"5cd6472dd826002cd3a214491565013c069a914f628ff5d2d88202a27e76f532"} Oct 30 06:59:03.461994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.461523 5001 scope.go:122] "RemoveContainer" containerID="8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab" Oct 30 06:59:03.462079 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:03.462048 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab\": container with ID starting with 8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab not found: ID does not exist" containerID="8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab" Oct 30 06:59:03.462141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.462093 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab"} err="failed to get container status \"8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab\": rpc error: code = NotFound desc = could not find container \"8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab\": container with ID starting with 8dd64699b33670fedb08193a27d7af2a0e14ae3f36df7e077e1ce79cdf78a7ab not found: ID does not exist" Oct 30 06:59:03.462141 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.462118 5001 scope.go:122] "RemoveContainer" containerID="7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204" Oct 30 06:59:03.473508 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.473485 5001 scope.go:122] "RemoveContainer" containerID="7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204" Oct 30 06:59:03.473792 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:03.473771 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204\": container with ID starting with 7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204 not found: ID does not exist" containerID="7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204" Oct 30 06:59:03.473863 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.473800 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204"} err="failed to get container status \"7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204\": rpc error: code = NotFound desc = could not find container \"7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204\": container with ID starting with 7dcf4a2d24b6088decbdacbf2c0bd6f24bd8d735c3bbc27eddd79d4b86aef204 not found: ID does not exist" Oct 30 06:59:03.531452 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.531414 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v"] Oct 30 06:59:03.539409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.539375 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr"] Oct 30 06:59:03.539658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.539632 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" containerID="cri-o://4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18" gracePeriod=30 Oct 30 06:59:03.542434 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.542405 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b8ca49-6bf6ffdcc9-zc52v"] Oct 30 06:59:03.581808 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.581771 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b"] Oct 30 06:59:03.589883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.589850 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b8ca49-7d87ddcdb6-ncd8b"] Oct 30 06:59:03.761640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:03.761534 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" podUID="813c0acd-5bc2-42ee-9eac-133d057df75f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:04.059895 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.059793 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl"] Oct 30 06:59:04.072184 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.072147 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc"] Oct 30 06:59:04.075767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.075728 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3a8846-9d8dl"] Oct 30 06:59:04.078830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.078808 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1740849-kkjnc"] Oct 30 06:59:04.117535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.117502 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39709b3b-ac31-48ce-8399-6c59101d4a32" path="/var/lib/kubelet/pods/39709b3b-ac31-48ce-8399-6c59101d4a32/volumes" Oct 30 06:59:04.117968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.117954 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53abaaf4-3338-48a9-8834-a24490e47beb" path="/var/lib/kubelet/pods/53abaaf4-3338-48a9-8834-a24490e47beb/volumes" Oct 30 06:59:04.118328 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.118295 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f1a036-b3e2-44e7-a082-862bf2070478" path="/var/lib/kubelet/pods/63f1a036-b3e2-44e7-a082-862bf2070478/volumes" Oct 30 06:59:04.119006 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.118991 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65ca078-eaa5-4bf4-a8c1-0856876434f1" path="/var/lib/kubelet/pods/b65ca078-eaa5-4bf4-a8c1-0856876434f1/volumes" Oct 30 06:59:04.179158 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.179112 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5"] Oct 30 06:59:04.179760 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.179747 5001 cpu_manager.go:414] "containerMap: removing container" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" Oct 30 06:59:04.179813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.179763 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" Oct 30 06:59:04.179813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.179789 5001 cpu_manager.go:414] "containerMap: removing container" podUID="63f1a036-b3e2-44e7-a082-862bf2070478" containerName="payload-processing" Oct 30 06:59:04.179813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.179794 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f1a036-b3e2-44e7-a082-862bf2070478" containerName="payload-processing" Oct 30 06:59:04.179813 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.179813 5001 cpu_manager.go:414] "containerMap: removing container" podUID="b65ca078-eaa5-4bf4-a8c1-0856876434f1" containerName="payload-pre-processing" Oct 30 06:59:04.179961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.179818 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65ca078-eaa5-4bf4-a8c1-0856876434f1" containerName="payload-pre-processing" Oct 30 06:59:04.179961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.179917 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd760fe2-e8c7-42f9-8a3f-40ec810159b4" containerName="istio-proxy" Oct 30 06:59:04.179961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.179930 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f1a036-b3e2-44e7-a082-862bf2070478" containerName="payload-processing" Oct 30 06:59:04.179961 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.179939 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65ca078-eaa5-4bf4-a8c1-0856876434f1" containerName="payload-pre-processing" Oct 30 06:59:04.185173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.185151 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:04.203946 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.203916 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5"] Oct 30 06:59:04.273599 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.273560 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vp2c\" (UniqueName: \"kubernetes.io/projected/498fd257-0dc2-413a-8d6f-265f468442f7-kube-api-access-7vp2c\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:04.273777 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.273630 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:04.273853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.273822 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:04.374839 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.374807 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:04.375036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.374907 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:04.375036 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.374990 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vp2c\" (UniqueName: \"kubernetes.io/projected/498fd257-0dc2-413a-8d6f-265f468442f7-kube-api-access-7vp2c\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:04.375622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.375598 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:04.377525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.377504 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:04.398520 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.398490 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vp2c\" (UniqueName: \"kubernetes.io/projected/498fd257-0dc2-413a-8d6f-265f468442f7-kube-api-access-7vp2c\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:04.499965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.499926 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:04.650748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.650670 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:04.827167 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.827129 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-xks-ca\") pod \"4d515235-aded-4643-a35e-e9c09fe86663\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " Oct 30 06:59:04.827389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.827201 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d515235-aded-4643-a35e-e9c09fe86663-kube-api-access-gsrgz\" (UniqueName: \"kubernetes.io/projected/4d515235-aded-4643-a35e-e9c09fe86663-kube-api-access-gsrgz\") pod \"4d515235-aded-4643-a35e-e9c09fe86663\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " Oct 30 06:59:04.827389 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.827267 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-openshift-ca\") pod \"4d515235-aded-4643-a35e-e9c09fe86663\" (UID: \"4d515235-aded-4643-a35e-e9c09fe86663\") " Oct 30 06:59:04.827654 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.827626 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-openshift-ca" pod "4d515235-aded-4643-a35e-e9c09fe86663" (UID: "4d515235-aded-4643-a35e-e9c09fe86663"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:04.827792 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.827783 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:04.830583 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.830557 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5"] Oct 30 06:59:04.830896 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.830866 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d515235-aded-4643-a35e-e9c09fe86663-kube-api-access-gsrgz" pod "4d515235-aded-4643-a35e-e9c09fe86663" (UID: "4d515235-aded-4643-a35e-e9c09fe86663"). InnerVolumeSpecName "kube-api-access-gsrgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:04.831937 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:59:04.831914 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498fd257_0dc2_413a_8d6f_265f468442f7.slice/crio-1d4b485ddcd3184bc7c9335f4f1c184d3ae0be8bed52da6a409263cd7e0a2280 WatchSource:0}: Error finding container 1d4b485ddcd3184bc7c9335f4f1c184d3ae0be8bed52da6a409263cd7e0a2280: Status 404 returned error can't find the container with id 1d4b485ddcd3184bc7c9335f4f1c184d3ae0be8bed52da6a409263cd7e0a2280 Oct 30 06:59:04.832853 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.832663 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-xks-ca" pod "4d515235-aded-4643-a35e-e9c09fe86663" (UID: "4d515235-aded-4643-a35e-e9c09fe86663"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:04.937404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.933550 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d515235-aded-4643-a35e-e9c09fe86663-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:04.937404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:04.933601 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsrgz\" (UniqueName: \"kubernetes.io/projected/4d515235-aded-4643-a35e-e9c09fe86663-kube-api-access-gsrgz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:05.007123 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:05.007080 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" podUID="e15ede9e-3c58-48c1-a663-1299268d6838" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:05.007357 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:05.007100 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" podUID="c553d6a6-04ad-43a5-abbd-73889c0797bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:05.265805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:05.265717 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9"] Oct 30 06:59:05.275507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:05.269759 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6d1778-tlsx9"] Oct 30 06:59:05.471685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:05.471637 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" event={"ID":"4d515235-aded-4643-a35e-e9c09fe86663","Type":"ContainerDied","Data":"91255ff5ebba0518e6c6492ad983a192170b48f8b7cffbf9c7b1e9a87a9f34fd"} Oct 30 06:59:05.471685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:05.471670 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk" Oct 30 06:59:05.471933 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:05.471702 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91255ff5ebba0518e6c6492ad983a192170b48f8b7cffbf9c7b1e9a87a9f34fd" Oct 30 06:59:05.473559 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:05.473529 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" event={"ID":"498fd257-0dc2-413a-8d6f-265f468442f7","Type":"ContainerStarted","Data":"cda2e10c50f8da63327dcb1bf33955857c0feb2f3897fe29487374dc51cb2733"} Oct 30 06:59:05.473691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:05.473570 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" event={"ID":"498fd257-0dc2-413a-8d6f-265f468442f7","Type":"ContainerStarted","Data":"1d4b485ddcd3184bc7c9335f4f1c184d3ae0be8bed52da6a409263cd7e0a2280"} Oct 30 06:59:05.514528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:05.514477 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" podStartSLOduration=1.514459761 podStartE2EDuration="1.514459761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:59:05.511945253 +0000 UTC m=+3959.507902264" watchObservedRunningTime="2026-10-30 06:59:05.514459761 +0000 UTC m=+3959.510416765" Oct 30 06:59:06.118968 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:06.118938 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc83cc44-158d-4461-913b-9070e289ae17" path="/var/lib/kubelet/pods/dc83cc44-158d-4461-913b-9070e289ae17/volumes" Oct 30 06:59:06.487911 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:06.487821 5001 generic.go:356] "Generic (PLEG): container finished" podID="498fd257-0dc2-413a-8d6f-265f468442f7" containerID="cda2e10c50f8da63327dcb1bf33955857c0feb2f3897fe29487374dc51cb2733" exitCode=0 Oct 30 06:59:06.488072 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:06.487897 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" event={"ID":"498fd257-0dc2-413a-8d6f-265f468442f7","Type":"ContainerDied","Data":"cda2e10c50f8da63327dcb1bf33955857c0feb2f3897fe29487374dc51cb2733"} Oct 30 06:59:07.683907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:07.683863 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:07.871564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:07.871525 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-openshift-ca\") pod \"498fd257-0dc2-413a-8d6f-265f468442f7\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " Oct 30 06:59:07.871767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:07.871587 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/498fd257-0dc2-413a-8d6f-265f468442f7-kube-api-access-7vp2c\" (UniqueName: \"kubernetes.io/projected/498fd257-0dc2-413a-8d6f-265f468442f7-kube-api-access-7vp2c\") pod \"498fd257-0dc2-413a-8d6f-265f468442f7\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " Oct 30 06:59:07.871767 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:07.871685 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-xks-ca\") pod \"498fd257-0dc2-413a-8d6f-265f468442f7\" (UID: \"498fd257-0dc2-413a-8d6f-265f468442f7\") " Oct 30 06:59:07.871944 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:07.871917 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-openshift-ca" pod "498fd257-0dc2-413a-8d6f-265f468442f7" (UID: "498fd257-0dc2-413a-8d6f-265f468442f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:07.872200 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:07.872177 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:07.874091 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:07.874059 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-xks-ca" pod "498fd257-0dc2-413a-8d6f-265f468442f7" (UID: "498fd257-0dc2-413a-8d6f-265f468442f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:07.874201 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:07.874100 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498fd257-0dc2-413a-8d6f-265f468442f7-kube-api-access-7vp2c" pod "498fd257-0dc2-413a-8d6f-265f468442f7" (UID: "498fd257-0dc2-413a-8d6f-265f468442f7"). InnerVolumeSpecName "kube-api-access-7vp2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:07.973352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:07.973321 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vp2c\" (UniqueName: \"kubernetes.io/projected/498fd257-0dc2-413a-8d6f-265f468442f7-kube-api-access-7vp2c\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:07.973352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:07.973349 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498fd257-0dc2-413a-8d6f-265f468442f7-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:08.265809 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:08.265774 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:59:08.512810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:08.512727 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" Oct 30 06:59:08.512810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:08.512753 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5" event={"ID":"498fd257-0dc2-413a-8d6f-265f468442f7","Type":"ContainerDied","Data":"1d4b485ddcd3184bc7c9335f4f1c184d3ae0be8bed52da6a409263cd7e0a2280"} Oct 30 06:59:08.512810 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:08.512802 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d4b485ddcd3184bc7c9335f4f1c184d3ae0be8bed52da6a409263cd7e0a2280" Oct 30 06:59:08.840643 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:08.840617 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:59:09.024326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.014251 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-certs\") pod \"00363ae6-95cd-4372-9209-487f572af4a9\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " Oct 30 06:59:09.024326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.014460 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-data\") pod \"00363ae6-95cd-4372-9209-487f572af4a9\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " Oct 30 06:59:09.024326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.014726 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-istio-token\" (UniqueName: \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-istio-token\") pod \"00363ae6-95cd-4372-9209-487f572af4a9\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " Oct 30 06:59:09.024326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.014913 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00363ae6-95cd-4372-9209-487f572af4a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00363ae6-95cd-4372-9209-487f572af4a9-istiod-ca-cert\") pod \"00363ae6-95cd-4372-9209-487f572af4a9\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " Oct 30 06:59:09.024326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.015061 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-certs" pod "00363ae6-95cd-4372-9209-487f572af4a9" (UID: "00363ae6-95cd-4372-9209-487f572af4a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:09.024326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.015127 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-envoy\") pod \"00363ae6-95cd-4372-9209-487f572af4a9\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " Oct 30 06:59:09.024326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.015345 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00363ae6-95cd-4372-9209-487f572af4a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00363ae6-95cd-4372-9209-487f572af4a9-istio-podinfo\") pod \"00363ae6-95cd-4372-9209-487f572af4a9\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " Oct 30 06:59:09.024326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.015529 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-kube-api-access-lhscv\" (UniqueName: \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-kube-api-access-lhscv\") pod \"00363ae6-95cd-4372-9209-487f572af4a9\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " Oct 30 06:59:09.024326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.015626 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-data" pod "00363ae6-95cd-4372-9209-487f572af4a9" (UID: "00363ae6-95cd-4372-9209-487f572af4a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:09.024326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.015676 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-credential-socket\") pod \"00363ae6-95cd-4372-9209-487f572af4a9\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " Oct 30 06:59:09.024326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.015721 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-socket\") pod \"00363ae6-95cd-4372-9209-487f572af4a9\" (UID: \"00363ae6-95cd-4372-9209-487f572af4a9\") " Oct 30 06:59:09.033084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.032754 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.033084 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.033041 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-socket" pod "00363ae6-95cd-4372-9209-487f572af4a9" (UID: "00363ae6-95cd-4372-9209-487f572af4a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:09.034069 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.034031 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00363ae6-95cd-4372-9209-487f572af4a9-istiod-ca-cert" pod "00363ae6-95cd-4372-9209-487f572af4a9" (UID: "00363ae6-95cd-4372-9209-487f572af4a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:09.035359 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.035338 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-credential-socket" pod "00363ae6-95cd-4372-9209-487f572af4a9" (UID: "00363ae6-95cd-4372-9209-487f572af4a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:09.038022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.037999 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00363ae6-95cd-4372-9209-487f572af4a9-istio-podinfo" pod "00363ae6-95cd-4372-9209-487f572af4a9" (UID: "00363ae6-95cd-4372-9209-487f572af4a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:59:09.038230 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.038209 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-kube-api-access-lhscv" pod "00363ae6-95cd-4372-9209-487f572af4a9" (UID: "00363ae6-95cd-4372-9209-487f572af4a9"). InnerVolumeSpecName "kube-api-access-lhscv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:09.038664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.038632 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-istio-token" pod "00363ae6-95cd-4372-9209-487f572af4a9" (UID: "00363ae6-95cd-4372-9209-487f572af4a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:09.039726 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.039687 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-envoy" pod "00363ae6-95cd-4372-9209-487f572af4a9" (UID: "00363ae6-95cd-4372-9209-487f572af4a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:09.133725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.133688 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.133725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.133716 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00363ae6-95cd-4372-9209-487f572af4a9-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.133725 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.133728 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.134016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.133740 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00363ae6-95cd-4372-9209-487f572af4a9-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.134016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.133749 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhscv\" (UniqueName: \"kubernetes.io/projected/00363ae6-95cd-4372-9209-487f572af4a9-kube-api-access-lhscv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.134016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.133757 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.134016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.133766 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.134016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.133774 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00363ae6-95cd-4372-9209-487f572af4a9-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.523780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.523698 5001 generic.go:356] "Generic (PLEG): container finished" podID="00363ae6-95cd-4372-9209-487f572af4a9" containerID="4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18" exitCode=0 Oct 30 06:59:09.523780 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.523755 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" Oct 30 06:59:09.524005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.523782 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" event={"ID":"00363ae6-95cd-4372-9209-487f572af4a9","Type":"ContainerDied","Data":"4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18"} Oct 30 06:59:09.524005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.523842 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr" event={"ID":"00363ae6-95cd-4372-9209-487f572af4a9","Type":"ContainerDied","Data":"80a7ca8a876e760382fdbcb64d0c16ebd175060165c3731d9d1e489b16758a96"} Oct 30 06:59:09.524005 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.523872 5001 scope.go:122] "RemoveContainer" containerID="4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18" Oct 30 06:59:09.539740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.539509 5001 scope.go:122] "RemoveContainer" containerID="7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d" Oct 30 06:59:09.552714 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.552691 5001 scope.go:122] "RemoveContainer" containerID="4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18" Oct 30 06:59:09.553069 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:09.553043 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18\": container with ID starting with 4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18 not found: ID does not exist" containerID="4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18" Oct 30 06:59:09.553175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.553082 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18"} err="failed to get container status \"4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18\": rpc error: code = NotFound desc = could not find container \"4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18\": container with ID starting with 4e5e647feec25f413e96d0dfddfaba485ca8d1eaabe48bd26db06cc7550acf18 not found: ID does not exist" Oct 30 06:59:09.553175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.553105 5001 scope.go:122] "RemoveContainer" containerID="7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d" Oct 30 06:59:09.553440 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:09.553420 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d\": container with ID starting with 7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d not found: ID does not exist" containerID="7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d" Oct 30 06:59:09.553518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.553448 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d"} err="failed to get container status \"7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d\": rpc error: code = NotFound desc = could not find container \"7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d\": container with ID starting with 7518f3b8acf870ff98eccd4b929c88583c00655329d62f4f786b86f92b80736d not found: ID does not exist" Oct 30 06:59:09.590851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.590818 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr"] Oct 30 06:59:09.602095 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:09.602065 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f0145f-openshift-default-6dc7d74d8f-6kgbr"] Oct 30 06:59:10.118499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:10.118468 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00363ae6-95cd-4372-9209-487f572af4a9" path="/var/lib/kubelet/pods/00363ae6-95cd-4372-9209-487f572af4a9/volumes" Oct 30 06:59:12.488635 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:12.488604 5001 scope.go:122] "RemoveContainer" containerID="09616e978afac430d6b02a74a1412b71a766f5ebf2a412c6043cf8b99261f195" Oct 30 06:59:12.503109 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:12.503085 5001 scope.go:122] "RemoveContainer" containerID="a91ebf922da226cc273e56be97d3512a9a2ccb89891cf1ea50ff188a3d6d487e" Oct 30 06:59:12.542001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:12.541969 5001 scope.go:122] "RemoveContainer" containerID="04e718a56ca683afa534828030465ec6e453adf36c28b9bc5da36e98e4910953" Oct 30 06:59:13.285627 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.284199 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:59:13.733485 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.733451 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk"] Oct 30 06:59:13.734085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734069 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4d515235-aded-4643-a35e-e9c09fe86663" containerName="revoke-keys" Oct 30 06:59:13.734085 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734085 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d515235-aded-4643-a35e-e9c09fe86663" containerName="revoke-keys" Oct 30 06:59:13.734233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734103 5001 cpu_manager.go:414] "containerMap: removing container" podUID="498fd257-0dc2-413a-8d6f-265f468442f7" containerName="revoke-keys" Oct 30 06:59:13.734233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734110 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="498fd257-0dc2-413a-8d6f-265f468442f7" containerName="revoke-keys" Oct 30 06:59:13.734233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734119 5001 cpu_manager.go:414] "containerMap: removing container" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" Oct 30 06:59:13.734233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734125 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" Oct 30 06:59:13.734233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734135 5001 cpu_manager.go:414] "containerMap: removing container" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" Oct 30 06:59:13.734233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734139 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" Oct 30 06:59:13.734233 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734229 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d515235-aded-4643-a35e-e9c09fe86663" containerName="revoke-keys" Oct 30 06:59:13.734642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734238 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="498fd257-0dc2-413a-8d6f-265f468442f7" containerName="revoke-keys" Oct 30 06:59:13.734642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734245 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" Oct 30 06:59:13.734642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.734251 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="00363ae6-95cd-4372-9209-487f572af4a9" containerName="istio-proxy" Oct 30 06:59:13.740014 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.739995 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:13.747098 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.747066 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" podUID="813c0acd-5bc2-42ee-9eac-133d057df75f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:13.754064 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.754035 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:59:13.758973 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.758944 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk"] Oct 30 06:59:13.764588 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.764560 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:59:13.764742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.764607 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:59:13.764742 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.764638 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:59:13.803669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.803635 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbkvv\" (UniqueName: \"kubernetes.io/projected/60e76528-1ac2-4b14-b467-6b1bd75a3e68-kube-api-access-pbkvv\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:13.803669 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.803674 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:13.803883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.803839 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:13.905243 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.905196 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:13.905484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.905423 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbkvv\" (UniqueName: \"kubernetes.io/projected/60e76528-1ac2-4b14-b467-6b1bd75a3e68-kube-api-access-pbkvv\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:13.905484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.905471 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:13.905939 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.905918 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:13.908145 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.908121 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:13.929129 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:13.929097 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbkvv\" (UniqueName: \"kubernetes.io/projected/60e76528-1ac2-4b14-b467-6b1bd75a3e68-kube-api-access-pbkvv\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:14.054057 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:14.053958 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:14.362341 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:14.362296 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk"] Oct 30 06:59:14.364813 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:59:14.364783 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e76528_1ac2_4b14_b467_6b1bd75a3e68.slice/crio-2cfe977c94b80e7267198a729de743fce23fe7f6382f62c83e422686bc8fbad9 WatchSource:0}: Error finding container 2cfe977c94b80e7267198a729de743fce23fe7f6382f62c83e422686bc8fbad9: Status 404 returned error can't find the container with id 2cfe977c94b80e7267198a729de743fce23fe7f6382f62c83e422686bc8fbad9 Oct 30 06:59:14.572676 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:14.572644 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" event={"ID":"60e76528-1ac2-4b14-b467-6b1bd75a3e68","Type":"ContainerStarted","Data":"2cfe977c94b80e7267198a729de743fce23fe7f6382f62c83e422686bc8fbad9"} Oct 30 06:59:14.632365 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:14.632274 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8"] Oct 30 06:59:14.665099 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:14.665071 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d"] Oct 30 06:59:15.007684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:15.007586 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" podUID="c553d6a6-04ad-43a5-abbd-73889c0797bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:15.007684 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:15.007618 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" podUID="e15ede9e-3c58-48c1-a663-1299268d6838" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:15.584245 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:15.584211 5001 generic.go:356] "Generic (PLEG): container finished" podID="60e76528-1ac2-4b14-b467-6b1bd75a3e68" containerID="ebee73531a682c2bc299e1b926b8b1c23f4bcfa398e83855a536727846da648a" exitCode=0 Oct 30 06:59:15.584443 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:15.584294 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" event={"ID":"60e76528-1ac2-4b14-b467-6b1bd75a3e68","Type":"ContainerDied","Data":"ebee73531a682c2bc299e1b926b8b1c23f4bcfa398e83855a536727846da648a"} Oct 30 06:59:16.757251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:16.757225 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:16.843750 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:16.843678 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-openshift-ca\") pod \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " Oct 30 06:59:16.843910 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:16.843784 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e76528-1ac2-4b14-b467-6b1bd75a3e68-kube-api-access-pbkvv\" (UniqueName: \"kubernetes.io/projected/60e76528-1ac2-4b14-b467-6b1bd75a3e68-kube-api-access-pbkvv\") pod \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " Oct 30 06:59:16.843975 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:16.843956 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-xks-ca\") pod \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\" (UID: \"60e76528-1ac2-4b14-b467-6b1bd75a3e68\") " Oct 30 06:59:16.844117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:16.844090 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-openshift-ca" pod "60e76528-1ac2-4b14-b467-6b1bd75a3e68" (UID: "60e76528-1ac2-4b14-b467-6b1bd75a3e68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:16.844555 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:16.844527 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:16.846083 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:16.846053 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e76528-1ac2-4b14-b467-6b1bd75a3e68-kube-api-access-pbkvv" pod "60e76528-1ac2-4b14-b467-6b1bd75a3e68" (UID: "60e76528-1ac2-4b14-b467-6b1bd75a3e68"). InnerVolumeSpecName "kube-api-access-pbkvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:16.846176 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:16.846122 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-xks-ca" pod "60e76528-1ac2-4b14-b467-6b1bd75a3e68" (UID: "60e76528-1ac2-4b14-b467-6b1bd75a3e68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:16.945841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:16.945811 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbkvv\" (UniqueName: \"kubernetes.io/projected/60e76528-1ac2-4b14-b467-6b1bd75a3e68-kube-api-access-pbkvv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:16.945841 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:16.945841 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e76528-1ac2-4b14-b467-6b1bd75a3e68-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.007054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.007027 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:59:17.046965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.046937 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813c0acd-5bc2-42ee-9eac-133d057df75f-kube-api-access-m5j9j\" (UniqueName: \"kubernetes.io/projected/813c0acd-5bc2-42ee-9eac-133d057df75f-kube-api-access-m5j9j\") pod \"813c0acd-5bc2-42ee-9eac-133d057df75f\" (UID: \"813c0acd-5bc2-42ee-9eac-133d057df75f\") " Oct 30 06:59:17.047117 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.047040 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/813c0acd-5bc2-42ee-9eac-133d057df75f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/813c0acd-5bc2-42ee-9eac-133d057df75f-plugins-config-volume\") pod \"813c0acd-5bc2-42ee-9eac-133d057df75f\" (UID: \"813c0acd-5bc2-42ee-9eac-133d057df75f\") " Oct 30 06:59:17.047719 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.047694 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813c0acd-5bc2-42ee-9eac-133d057df75f-plugins-config-volume" pod "813c0acd-5bc2-42ee-9eac-133d057df75f" (UID: "813c0acd-5bc2-42ee-9eac-133d057df75f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:17.050004 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.049966 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813c0acd-5bc2-42ee-9eac-133d057df75f-kube-api-access-m5j9j" pod "813c0acd-5bc2-42ee-9eac-133d057df75f" (UID: "813c0acd-5bc2-42ee-9eac-133d057df75f"). InnerVolumeSpecName "kube-api-access-m5j9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:17.149382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.149347 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5j9j\" (UniqueName: \"kubernetes.io/projected/813c0acd-5bc2-42ee-9eac-133d057df75f-kube-api-access-m5j9j\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.149382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.149385 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/813c0acd-5bc2-42ee-9eac-133d057df75f-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.602840 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.602807 5001 generic.go:356] "Generic (PLEG): container finished" podID="813c0acd-5bc2-42ee-9eac-133d057df75f" containerID="88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388" exitCode=0 Oct 30 06:59:17.603026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.602875 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" Oct 30 06:59:17.603026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.602885 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" event={"ID":"813c0acd-5bc2-42ee-9eac-133d057df75f","Type":"ContainerDied","Data":"88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388"} Oct 30 06:59:17.603026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.602971 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w" event={"ID":"813c0acd-5bc2-42ee-9eac-133d057df75f","Type":"ContainerDied","Data":"64fa9f7ac4f5999319a25a3824304126cf03b0f0cd7e7794e9cb86a094954d7a"} Oct 30 06:59:17.603026 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.602999 5001 scope.go:122] "RemoveContainer" containerID="88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388" Oct 30 06:59:17.604683 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.604666 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" Oct 30 06:59:17.604784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.604722 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk" event={"ID":"60e76528-1ac2-4b14-b467-6b1bd75a3e68","Type":"ContainerDied","Data":"2cfe977c94b80e7267198a729de743fce23fe7f6382f62c83e422686bc8fbad9"} Oct 30 06:59:17.604784 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.604760 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cfe977c94b80e7267198a729de743fce23fe7f6382f62c83e422686bc8fbad9" Oct 30 06:59:17.626385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.626364 5001 scope.go:122] "RemoveContainer" containerID="88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388" Oct 30 06:59:17.626782 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:17.626758 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388\": container with ID starting with 88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388 not found: ID does not exist" containerID="88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388" Oct 30 06:59:17.626905 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.626792 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388"} err="failed to get container status \"88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388\": rpc error: code = NotFound desc = could not find container \"88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388\": container with ID starting with 88f0e4728e2d2b836a7fc7b711750785fcdc130b9740faf78e6a7eb38a0cd388 not found: ID does not exist" Oct 30 06:59:17.658795 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.658748 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w"] Oct 30 06:59:17.672285 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.672247 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6d1778-649f5db87b-kqr6w"] Oct 30 06:59:17.799678 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.799643 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:59:17.859421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.859384 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c553d6a6-04ad-43a5-abbd-73889c0797bb-kube-api-access-j5p74\" (UniqueName: \"kubernetes.io/projected/c553d6a6-04ad-43a5-abbd-73889c0797bb-kube-api-access-j5p74\") pod \"c553d6a6-04ad-43a5-abbd-73889c0797bb\" (UID: \"c553d6a6-04ad-43a5-abbd-73889c0797bb\") " Oct 30 06:59:17.859609 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.859504 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c553d6a6-04ad-43a5-abbd-73889c0797bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c553d6a6-04ad-43a5-abbd-73889c0797bb-plugins-config-volume\") pod \"c553d6a6-04ad-43a5-abbd-73889c0797bb\" (UID: \"c553d6a6-04ad-43a5-abbd-73889c0797bb\") " Oct 30 06:59:17.860251 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.860196 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c553d6a6-04ad-43a5-abbd-73889c0797bb-plugins-config-volume" pod "c553d6a6-04ad-43a5-abbd-73889c0797bb" (UID: "c553d6a6-04ad-43a5-abbd-73889c0797bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:17.862751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.862729 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c553d6a6-04ad-43a5-abbd-73889c0797bb-kube-api-access-j5p74" pod "c553d6a6-04ad-43a5-abbd-73889c0797bb" (UID: "c553d6a6-04ad-43a5-abbd-73889c0797bb"). InnerVolumeSpecName "kube-api-access-j5p74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:17.872382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.872358 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:59:17.961059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.961020 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15ede9e-3c58-48c1-a663-1299268d6838-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ede9e-3c58-48c1-a663-1299268d6838-plugins-config-volume\") pod \"e15ede9e-3c58-48c1-a663-1299268d6838\" (UID: \"e15ede9e-3c58-48c1-a663-1299268d6838\") " Oct 30 06:59:17.961242 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.961155 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15ede9e-3c58-48c1-a663-1299268d6838-kube-api-access-mtpmx\" (UniqueName: \"kubernetes.io/projected/e15ede9e-3c58-48c1-a663-1299268d6838-kube-api-access-mtpmx\") pod \"e15ede9e-3c58-48c1-a663-1299268d6838\" (UID: \"e15ede9e-3c58-48c1-a663-1299268d6838\") " Oct 30 06:59:17.961424 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.961403 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15ede9e-3c58-48c1-a663-1299268d6838-plugins-config-volume" pod "e15ede9e-3c58-48c1-a663-1299268d6838" (UID: "e15ede9e-3c58-48c1-a663-1299268d6838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:17.961830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.961810 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5p74\" (UniqueName: \"kubernetes.io/projected/c553d6a6-04ad-43a5-abbd-73889c0797bb-kube-api-access-j5p74\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.961865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.961838 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ede9e-3c58-48c1-a663-1299268d6838-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.961865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.961855 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c553d6a6-04ad-43a5-abbd-73889c0797bb-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.963458 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:17.963428 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15ede9e-3c58-48c1-a663-1299268d6838-kube-api-access-mtpmx" pod "e15ede9e-3c58-48c1-a663-1299268d6838" (UID: "e15ede9e-3c58-48c1-a663-1299268d6838"). InnerVolumeSpecName "kube-api-access-mtpmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:18.062899 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.062812 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtpmx\" (UniqueName: \"kubernetes.io/projected/e15ede9e-3c58-48c1-a663-1299268d6838-kube-api-access-mtpmx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:18.120380 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.120343 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813c0acd-5bc2-42ee-9eac-133d057df75f" path="/var/lib/kubelet/pods/813c0acd-5bc2-42ee-9eac-133d057df75f/volumes" Oct 30 06:59:18.614875 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.614837 5001 generic.go:356] "Generic (PLEG): container finished" podID="e15ede9e-3c58-48c1-a663-1299268d6838" containerID="db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164" exitCode=0 Oct 30 06:59:18.615101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.614880 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" Oct 30 06:59:18.615101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.614919 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" event={"ID":"e15ede9e-3c58-48c1-a663-1299268d6838","Type":"ContainerDied","Data":"db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164"} Oct 30 06:59:18.615101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.614977 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x" event={"ID":"e15ede9e-3c58-48c1-a663-1299268d6838","Type":"ContainerDied","Data":"ae6fbac0b65a327f75c340c51af4126505f2cd2239551560ccc9054e86945c9e"} Oct 30 06:59:18.615101 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.615003 5001 scope.go:122] "RemoveContainer" containerID="db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164" Oct 30 06:59:18.617033 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.617007 5001 generic.go:356] "Generic (PLEG): container finished" podID="c553d6a6-04ad-43a5-abbd-73889c0797bb" containerID="01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e" exitCode=0 Oct 30 06:59:18.617160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.617061 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" event={"ID":"c553d6a6-04ad-43a5-abbd-73889c0797bb","Type":"ContainerDied","Data":"01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e"} Oct 30 06:59:18.617160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.617130 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" event={"ID":"c553d6a6-04ad-43a5-abbd-73889c0797bb","Type":"ContainerDied","Data":"0561db3827a409a8f41a9a089d50507d1aa328abdafe553bfc434262cad987e9"} Oct 30 06:59:18.617160 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.617071 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4" Oct 30 06:59:18.630196 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.630172 5001 scope.go:122] "RemoveContainer" containerID="db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164" Oct 30 06:59:18.630499 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:18.630471 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164\": container with ID starting with db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164 not found: ID does not exist" containerID="db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164" Oct 30 06:59:18.630569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.630501 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164"} err="failed to get container status \"db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164\": rpc error: code = NotFound desc = could not find container \"db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164\": container with ID starting with db5340a88d9b5a79bb171bde977689fc8fbf696ed8954d81920e9a63d168f164 not found: ID does not exist" Oct 30 06:59:18.630569 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.630517 5001 scope.go:122] "RemoveContainer" containerID="01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e" Oct 30 06:59:18.641340 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.641289 5001 scope.go:122] "RemoveContainer" containerID="01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e" Oct 30 06:59:18.641611 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:18.641592 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e\": container with ID starting with 01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e not found: ID does not exist" containerID="01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e" Oct 30 06:59:18.641673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.641618 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e"} err="failed to get container status \"01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e\": rpc error: code = NotFound desc = could not find container \"01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e\": container with ID starting with 01793a086772d2eb499f589554ad93624f8955bdf9e0e5612da7cfc430a47f2e not found: ID does not exist" Oct 30 06:59:18.667529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.667495 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x"] Oct 30 06:59:18.671393 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.671366 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f0145f-65b548dbc-mp68x"] Oct 30 06:59:18.704065 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.704028 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4"] Oct 30 06:59:18.708111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:18.708083 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f0145f-78879f4dd-8cbb4"] Oct 30 06:59:19.632924 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:19.632880 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" podUID="75d360de-4c0c-4d3f-bc06-1ea1922e9f43" containerName="payload-processing" containerID="cri-o://2cd7ebec86e493d6bb27627fba5bc8185d9d9b30775015808af77ecb44386432" gracePeriod=25 Oct 30 06:59:19.665625 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:19.665582 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" podUID="8efc4083-009c-45f8-b535-7dff63289742" containerName="payload-pre-processing" containerID="cri-o://8d2484a8a870f627e85ae4b37fb447e3ee44733afbcded16021dd4e8aa4f108c" gracePeriod=25 Oct 30 06:59:20.117378 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:20.117345 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c553d6a6-04ad-43a5-abbd-73889c0797bb" path="/var/lib/kubelet/pods/c553d6a6-04ad-43a5-abbd-73889c0797bb/volumes" Oct 30 06:59:20.117747 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:20.117734 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15ede9e-3c58-48c1-a663-1299268d6838" path="/var/lib/kubelet/pods/e15ede9e-3c58-48c1-a663-1299268d6838/volumes" Oct 30 06:59:20.253957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:20.253922 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5"] Oct 30 06:59:20.265386 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:20.265354 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d8df2-hffm5"] Oct 30 06:59:20.802731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:20.802689 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk"] Oct 30 06:59:20.813971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:20.813941 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-69f3fa-kmpgk"] Oct 30 06:59:22.118130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:22.118096 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="498fd257-0dc2-413a-8d6f-265f468442f7" path="/var/lib/kubelet/pods/498fd257-0dc2-413a-8d6f-265f468442f7/volumes" Oct 30 06:59:22.118606 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:22.118588 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d515235-aded-4643-a35e-e9c09fe86663" path="/var/lib/kubelet/pods/4d515235-aded-4643-a35e-e9c09fe86663/volumes" Oct 30 06:59:24.174360 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:24.174324 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m"] Oct 30 06:59:27.470027 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:27.469986 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" podUID="8efc4083-009c-45f8-b535-7dff63289742" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:27.470673 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:27.470025 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" podUID="75d360de-4c0c-4d3f-bc06-1ea1922e9f43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:27.860435 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:27.860401 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2"] Oct 30 06:59:27.860685 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:27.860655 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" containerID="cri-o://583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78" gracePeriod=30 Oct 30 06:59:28.486472 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.486436 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4"] Oct 30 06:59:28.486998 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.486985 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c553d6a6-04ad-43a5-abbd-73889c0797bb" containerName="payload-processing" Oct 30 06:59:28.487054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.486999 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c553d6a6-04ad-43a5-abbd-73889c0797bb" containerName="payload-processing" Oct 30 06:59:28.487054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.487022 5001 cpu_manager.go:414] "containerMap: removing container" podUID="60e76528-1ac2-4b14-b467-6b1bd75a3e68" containerName="revoke-keys" Oct 30 06:59:28.487054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.487027 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e76528-1ac2-4b14-b467-6b1bd75a3e68" containerName="revoke-keys" Oct 30 06:59:28.487054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.487037 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e15ede9e-3c58-48c1-a663-1299268d6838" containerName="payload-pre-processing" Oct 30 06:59:28.487054 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.487043 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15ede9e-3c58-48c1-a663-1299268d6838" containerName="payload-pre-processing" Oct 30 06:59:28.487228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.487086 5001 cpu_manager.go:414] "containerMap: removing container" podUID="813c0acd-5bc2-42ee-9eac-133d057df75f" containerName="payload-processing" Oct 30 06:59:28.487228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.487092 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813c0acd-5bc2-42ee-9eac-133d057df75f" containerName="payload-processing" Oct 30 06:59:28.487228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.487178 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="813c0acd-5bc2-42ee-9eac-133d057df75f" containerName="payload-processing" Oct 30 06:59:28.487228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.487187 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c553d6a6-04ad-43a5-abbd-73889c0797bb" containerName="payload-processing" Oct 30 06:59:28.487228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.487196 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e76528-1ac2-4b14-b467-6b1bd75a3e68" containerName="revoke-keys" Oct 30 06:59:28.487228 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.487202 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15ede9e-3c58-48c1-a663-1299268d6838" containerName="payload-pre-processing" Oct 30 06:59:28.492621 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.492600 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:28.514867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.513832 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:59:28.514867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.513914 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:59:28.514867 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.513974 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:59:28.525542 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.525510 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:59:28.530636 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.530604 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4"] Oct 30 06:59:28.619175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.619127 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:28.619175 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.619187 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47bqm\" (UniqueName: \"kubernetes.io/projected/679edb69-a667-4c6f-856e-4b62fb069d63-kube-api-access-47bqm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:28.619453 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.619317 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:28.720597 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.720562 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:28.720822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.720620 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47bqm\" (UniqueName: \"kubernetes.io/projected/679edb69-a667-4c6f-856e-4b62fb069d63-kube-api-access-47bqm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:28.720822 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.720679 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:28.721281 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.721254 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:28.723156 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.723135 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:28.742965 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.742906 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47bqm\" (UniqueName: \"kubernetes.io/projected/679edb69-a667-4c6f-856e-4b62fb069d63-kube-api-access-47bqm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:28.792506 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.792466 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g"] Oct 30 06:59:28.792846 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.792798 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" containerID="cri-o://ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4" gracePeriod=30 Oct 30 06:59:28.809020 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:28.808998 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:29.119092 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.119067 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4"] Oct 30 06:59:29.121158 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:59:29.121127 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679edb69_a667_4c6f_856e_4b62fb069d63.slice/crio-f1c43b3afe8b9f49fd73bb901d269641f2a5966edeb0eae6376902b4a35509c6 WatchSource:0}: Error finding container f1c43b3afe8b9f49fd73bb901d269641f2a5966edeb0eae6376902b4a35509c6: Status 404 returned error can't find the container with id f1c43b3afe8b9f49fd73bb901d269641f2a5966edeb0eae6376902b4a35509c6 Oct 30 06:59:29.175155 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.175111 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" podUID="dd5eddac-dbe2-485e-8c7b-1fec89d02eeb" containerName="payload-processing" containerID="cri-o://596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974" gracePeriod=25 Oct 30 06:59:29.418299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.418217 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8"] Oct 30 06:59:29.423932 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.423911 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:29.439791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.439595 5001 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:59:29.439791 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.439667 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:59:29.442012 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.441989 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8"] Oct 30 06:59:29.537239 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.537206 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcwdz\" (UniqueName: \"kubernetes.io/projected/f9a8053a-ef53-4211-af13-dd7cc881cc91-kube-api-access-lcwdz\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:29.537642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.537266 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:29.537642 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.537387 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:29.638664 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.638631 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcwdz\" (UniqueName: \"kubernetes.io/projected/f9a8053a-ef53-4211-af13-dd7cc881cc91-kube-api-access-lcwdz\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:29.638866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.638700 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:29.638866 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.638735 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:29.639526 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.639503 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:29.641259 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.641236 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:29.661025 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.660992 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcwdz\" (UniqueName: \"kubernetes.io/projected/f9a8053a-ef53-4211-af13-dd7cc881cc91-kube-api-access-lcwdz\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:29.729799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.729711 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" event={"ID":"679edb69-a667-4c6f-856e-4b62fb069d63","Type":"ContainerStarted","Data":"14011bbfb0578adb8566b864b2d7931c64c0a619b67a25a803280e7a1c49ca85"} Oct 30 06:59:29.729799 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.729769 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" event={"ID":"679edb69-a667-4c6f-856e-4b62fb069d63","Type":"ContainerStarted","Data":"f1c43b3afe8b9f49fd73bb901d269641f2a5966edeb0eae6376902b4a35509c6"} Oct 30 06:59:29.738146 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.738123 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:29.768157 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:29.768103 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" podStartSLOduration=1.768085155 podStartE2EDuration="1.768085155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:59:29.767946584 +0000 UTC m=+3983.763903596" watchObservedRunningTime="2026-10-30 06:59:29.768085155 +0000 UTC m=+3983.764042166" Oct 30 06:59:30.056835 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:30.056801 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8"] Oct 30 06:59:30.058896 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:59:30.058869 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a8053a_ef53_4211_af13_dd7cc881cc91.slice/crio-7107baa762bd9c4ae859bbc5f4c968cd9d92b31236e1c0a813a12fc7bab45abf WatchSource:0}: Error finding container 7107baa762bd9c4ae859bbc5f4c968cd9d92b31236e1c0a813a12fc7bab45abf: Status 404 returned error can't find the container with id 7107baa762bd9c4ae859bbc5f4c968cd9d92b31236e1c0a813a12fc7bab45abf Oct 30 06:59:30.739525 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:30.739495 5001 generic.go:356] "Generic (PLEG): container finished" podID="679edb69-a667-4c6f-856e-4b62fb069d63" containerID="14011bbfb0578adb8566b864b2d7931c64c0a619b67a25a803280e7a1c49ca85" exitCode=0 Oct 30 06:59:30.739925 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:30.739581 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" event={"ID":"679edb69-a667-4c6f-856e-4b62fb069d63","Type":"ContainerDied","Data":"14011bbfb0578adb8566b864b2d7931c64c0a619b67a25a803280e7a1c49ca85"} Oct 30 06:59:30.741437 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:30.741412 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" event={"ID":"f9a8053a-ef53-4211-af13-dd7cc881cc91","Type":"ContainerStarted","Data":"aba03d595ea67b76bc840e2fd1ac17f9c56d5fb2696d01d72056fd25f61afdee"} Oct 30 06:59:30.741572 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:30.741455 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" event={"ID":"f9a8053a-ef53-4211-af13-dd7cc881cc91","Type":"ContainerStarted","Data":"7107baa762bd9c4ae859bbc5f4c968cd9d92b31236e1c0a813a12fc7bab45abf"} Oct 30 06:59:30.820931 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:30.820880 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" podStartSLOduration=1.820861511 podStartE2EDuration="1.820861511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:59:30.818043395 +0000 UTC m=+3984.814000410" watchObservedRunningTime="2026-10-30 06:59:30.820861511 +0000 UTC m=+3984.816818518" Oct 30 06:59:30.962366 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:30.962328 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:59:31.762046 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:31.762008 5001 generic.go:356] "Generic (PLEG): container finished" podID="f9a8053a-ef53-4211-af13-dd7cc881cc91" containerID="aba03d595ea67b76bc840e2fd1ac17f9c56d5fb2696d01d72056fd25f61afdee" exitCode=0 Oct 30 06:59:31.762549 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:31.762103 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" event={"ID":"f9a8053a-ef53-4211-af13-dd7cc881cc91","Type":"ContainerDied","Data":"aba03d595ea67b76bc840e2fd1ac17f9c56d5fb2696d01d72056fd25f61afdee"} Oct 30 06:59:31.919001 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:31.918976 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:31.969696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:31.969665 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679edb69-a667-4c6f-856e-4b62fb069d63-kube-api-access-47bqm\" (UniqueName: \"kubernetes.io/projected/679edb69-a667-4c6f-856e-4b62fb069d63-kube-api-access-47bqm\") pod \"679edb69-a667-4c6f-856e-4b62fb069d63\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " Oct 30 06:59:31.969906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:31.969760 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-xks-ca\") pod \"679edb69-a667-4c6f-856e-4b62fb069d63\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " Oct 30 06:59:31.969906 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:31.969814 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-openshift-ca\") pod \"679edb69-a667-4c6f-856e-4b62fb069d63\" (UID: \"679edb69-a667-4c6f-856e-4b62fb069d63\") " Oct 30 06:59:31.970202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:31.970171 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-openshift-ca" pod "679edb69-a667-4c6f-856e-4b62fb069d63" (UID: "679edb69-a667-4c6f-856e-4b62fb069d63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:31.972180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:31.972153 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-xks-ca" pod "679edb69-a667-4c6f-856e-4b62fb069d63" (UID: "679edb69-a667-4c6f-856e-4b62fb069d63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:31.972292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:31.972216 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679edb69-a667-4c6f-856e-4b62fb069d63-kube-api-access-47bqm" pod "679edb69-a667-4c6f-856e-4b62fb069d63" (UID: "679edb69-a667-4c6f-856e-4b62fb069d63"). InnerVolumeSpecName "kube-api-access-47bqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:32.071404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:32.071295 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47bqm\" (UniqueName: \"kubernetes.io/projected/679edb69-a667-4c6f-856e-4b62fb069d63-kube-api-access-47bqm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:32.071404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:32.071342 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:32.071404 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:32.071352 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/679edb69-a667-4c6f-856e-4b62fb069d63-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:32.772376 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:32.772345 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" Oct 30 06:59:32.772859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:32.772348 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4" event={"ID":"679edb69-a667-4c6f-856e-4b62fb069d63","Type":"ContainerDied","Data":"f1c43b3afe8b9f49fd73bb901d269641f2a5966edeb0eae6376902b4a35509c6"} Oct 30 06:59:32.772859 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:32.772486 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1c43b3afe8b9f49fd73bb901d269641f2a5966edeb0eae6376902b4a35509c6" Oct 30 06:59:32.993125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:32.993097 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:33.083935 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.083884 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a8053a-ef53-4211-af13-dd7cc881cc91-kube-api-access-lcwdz\" (UniqueName: \"kubernetes.io/projected/f9a8053a-ef53-4211-af13-dd7cc881cc91-kube-api-access-lcwdz\") pod \"f9a8053a-ef53-4211-af13-dd7cc881cc91\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " Oct 30 06:59:33.084130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.084016 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-openshift-ca\") pod \"f9a8053a-ef53-4211-af13-dd7cc881cc91\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " Oct 30 06:59:33.084195 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.084135 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-xks-ca\") pod \"f9a8053a-ef53-4211-af13-dd7cc881cc91\" (UID: \"f9a8053a-ef53-4211-af13-dd7cc881cc91\") " Oct 30 06:59:33.084696 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.084666 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-openshift-ca" pod "f9a8053a-ef53-4211-af13-dd7cc881cc91" (UID: "f9a8053a-ef53-4211-af13-dd7cc881cc91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:33.090717 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.090689 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-xks-ca" pod "f9a8053a-ef53-4211-af13-dd7cc881cc91" (UID: "f9a8053a-ef53-4211-af13-dd7cc881cc91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:33.091662 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.091636 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a8053a-ef53-4211-af13-dd7cc881cc91-kube-api-access-lcwdz" pod "f9a8053a-ef53-4211-af13-dd7cc881cc91" (UID: "f9a8053a-ef53-4211-af13-dd7cc881cc91"). InnerVolumeSpecName "kube-api-access-lcwdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:33.126339 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.126291 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:59:33.185291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.185252 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-credential-socket\") pod \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " Oct 30 06:59:33.185474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.185347 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-data\") pod \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " Oct 30 06:59:33.185474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.185390 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-token\") pod \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " Oct 30 06:59:33.185474 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.185447 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istiod-ca-cert\") pod \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " Oct 30 06:59:33.185658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.185492 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-socket\") pod \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " Oct 30 06:59:33.185658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.185579 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-envoy\") pod \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " Oct 30 06:59:33.185658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.185586 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-credential-socket" pod "8b9fe738-2674-4d41-a255-1c2a057d4ca5" (UID: "8b9fe738-2674-4d41-a255-1c2a057d4ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:33.185658 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.185608 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-podinfo\") pod \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " Oct 30 06:59:33.185805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.185704 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-certs\") pod \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " Oct 30 06:59:33.185805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.185761 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-kube-api-access-w62sz\" (UniqueName: \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-kube-api-access-w62sz\") pod \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\" (UID: \"8b9fe738-2674-4d41-a255-1c2a057d4ca5\") " Oct 30 06:59:33.185929 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.185901 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istiod-ca-cert" pod "8b9fe738-2674-4d41-a255-1c2a057d4ca5" (UID: "8b9fe738-2674-4d41-a255-1c2a057d4ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:33.186173 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.186148 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-data" pod "8b9fe738-2674-4d41-a255-1c2a057d4ca5" (UID: "8b9fe738-2674-4d41-a255-1c2a057d4ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:33.186421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.186400 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.186534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.186428 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.186534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.186476 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a8053a-ef53-4211-af13-dd7cc881cc91-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.186534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.186489 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.186534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.186488 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-socket" pod "8b9fe738-2674-4d41-a255-1c2a057d4ca5" (UID: "8b9fe738-2674-4d41-a255-1c2a057d4ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:33.186534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.186501 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.186534 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.186514 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcwdz\" (UniqueName: \"kubernetes.io/projected/f9a8053a-ef53-4211-af13-dd7cc881cc91-kube-api-access-lcwdz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.187118 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.187093 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-certs" pod "8b9fe738-2674-4d41-a255-1c2a057d4ca5" (UID: "8b9fe738-2674-4d41-a255-1c2a057d4ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:33.188498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.188466 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-envoy" pod "8b9fe738-2674-4d41-a255-1c2a057d4ca5" (UID: "8b9fe738-2674-4d41-a255-1c2a057d4ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:33.189059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.189010 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-kube-api-access-w62sz" pod "8b9fe738-2674-4d41-a255-1c2a057d4ca5" (UID: "8b9fe738-2674-4d41-a255-1c2a057d4ca5"). InnerVolumeSpecName "kube-api-access-w62sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:33.189202 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.189109 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-podinfo" pod "8b9fe738-2674-4d41-a255-1c2a057d4ca5" (UID: "8b9fe738-2674-4d41-a255-1c2a057d4ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:59:33.189466 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.189446 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-token" pod "8b9fe738-2674-4d41-a255-1c2a057d4ca5" (UID: "8b9fe738-2674-4d41-a255-1c2a057d4ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:33.288111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.288031 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.288111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.288060 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.288111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.288069 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.288111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.288077 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9fe738-2674-4d41-a255-1c2a057d4ca5-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.288111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.288085 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9fe738-2674-4d41-a255-1c2a057d4ca5-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.288111 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.288098 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w62sz\" (UniqueName: \"kubernetes.io/projected/8b9fe738-2674-4d41-a255-1c2a057d4ca5-kube-api-access-w62sz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:33.783498 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.783459 5001 generic.go:356] "Generic (PLEG): container finished" podID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerID="583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78" exitCode=0 Oct 30 06:59:33.783955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.783517 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" Oct 30 06:59:33.783955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.783538 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" event={"ID":"8b9fe738-2674-4d41-a255-1c2a057d4ca5","Type":"ContainerDied","Data":"583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78"} Oct 30 06:59:33.783955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.783604 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2" event={"ID":"8b9fe738-2674-4d41-a255-1c2a057d4ca5","Type":"ContainerDied","Data":"cfb7df7bb8d02acaf2681d95a5990a33be02c89acaf25625d306ef199fefde90"} Oct 30 06:59:33.783955 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.783638 5001 scope.go:122] "RemoveContainer" containerID="583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78" Oct 30 06:59:33.785507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.785482 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" Oct 30 06:59:33.785674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.785527 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8" event={"ID":"f9a8053a-ef53-4211-af13-dd7cc881cc91","Type":"ContainerDied","Data":"7107baa762bd9c4ae859bbc5f4c968cd9d92b31236e1c0a813a12fc7bab45abf"} Oct 30 06:59:33.785674 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.785600 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7107baa762bd9c4ae859bbc5f4c968cd9d92b31236e1c0a813a12fc7bab45abf" Oct 30 06:59:33.799210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.799188 5001 scope.go:122] "RemoveContainer" containerID="10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3" Oct 30 06:59:33.811999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.811971 5001 scope.go:122] "RemoveContainer" containerID="583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78" Oct 30 06:59:33.812436 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:33.812408 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78\": container with ID starting with 583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78 not found: ID does not exist" containerID="583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78" Oct 30 06:59:33.812566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.812449 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78"} err="failed to get container status \"583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78\": rpc error: code = NotFound desc = could not find container \"583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78\": container with ID starting with 583a9a600f2f96dc5223ab2748ca41de5f9a039709e8c9bdd15b9f1ad841fe78 not found: ID does not exist" Oct 30 06:59:33.812566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.812474 5001 scope.go:122] "RemoveContainer" containerID="10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3" Oct 30 06:59:33.812887 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:33.812866 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3\": container with ID starting with 10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3 not found: ID does not exist" containerID="10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3" Oct 30 06:59:33.812949 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.812893 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3"} err="failed to get container status \"10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3\": rpc error: code = NotFound desc = could not find container \"10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3\": container with ID starting with 10a6cddc23715f4d14d68a8c0c7d84cd9869fd5e10822c60112e1399c7ad07e3 not found: ID does not exist" Oct 30 06:59:33.837060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.837025 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2"] Oct 30 06:59:33.848593 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:33.848559 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-2ktr2"] Oct 30 06:59:34.096528 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.096501 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:59:34.117721 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.117685 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" path="/var/lib/kubelet/pods/8b9fe738-2674-4d41-a255-1c2a057d4ca5/volumes" Oct 30 06:59:34.196612 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.196582 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-certs\") pod \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " Oct 30 06:59:34.196821 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.196700 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-token\" (UniqueName: \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-token\") pod \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " Oct 30 06:59:34.196876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.196831 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-envoy\") pod \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " Oct 30 06:59:34.196876 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.196867 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-data\") pod \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " Oct 30 06:59:34.196994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.196906 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-kube-api-access-w8rnv\" (UniqueName: \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-kube-api-access-w8rnv\") pod \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " Oct 30 06:59:34.196994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.196918 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-certs" pod "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" (UID: "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:34.196994 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.196965 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-podinfo\") pod \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " Oct 30 06:59:34.197180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.196993 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-socket\") pod \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " Oct 30 06:59:34.197180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.197032 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-credential-socket\") pod \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " Oct 30 06:59:34.197180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.197083 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istiod-ca-cert\") pod \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\" (UID: \"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78\") " Oct 30 06:59:34.197495 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.197464 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-socket" pod "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" (UID: "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:34.197622 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.197555 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-credential-socket" pod "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" (UID: "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:34.198203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.197826 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:34.198203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.197854 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:34.198203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.197871 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:34.198203 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.197945 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-data" pod "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" (UID: "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:34.198552 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.198250 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istiod-ca-cert" pod "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" (UID: "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:34.199422 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.199397 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-envoy" pod "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" (UID: "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:34.199645 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.199629 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-podinfo" pod "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" (UID: "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:59:34.199761 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.199743 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-token" pod "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" (UID: "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:34.199805 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.199770 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-kube-api-access-w8rnv" pod "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" (UID: "f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78"). InnerVolumeSpecName "kube-api-access-w8rnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:34.299081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.299037 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:34.299081 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.299080 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:34.299326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.299094 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:34.299326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.299108 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8rnv\" (UniqueName: \"kubernetes.io/projected/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-kube-api-access-w8rnv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:34.299326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.299126 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:34.299326 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.299139 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:34.797815 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.797780 5001 generic.go:356] "Generic (PLEG): container finished" podID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerID="ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4" exitCode=0 Oct 30 06:59:34.798344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.797837 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" Oct 30 06:59:34.798344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.797851 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" event={"ID":"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78","Type":"ContainerDied","Data":"ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4"} Oct 30 06:59:34.798344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.797892 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g" event={"ID":"f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78","Type":"ContainerDied","Data":"38ba3fd6feb388d8dbb2f71d5983d0dc37aa246ad731f909c64a83366430f82d"} Oct 30 06:59:34.798344 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.797916 5001 scope.go:122] "RemoveContainer" containerID="ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4" Oct 30 06:59:34.810843 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.810824 5001 scope.go:122] "RemoveContainer" containerID="d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c" Oct 30 06:59:34.821691 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.821661 5001 scope.go:122] "RemoveContainer" containerID="ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4" Oct 30 06:59:34.821963 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:34.821942 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4\": container with ID starting with ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4 not found: ID does not exist" containerID="ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4" Oct 30 06:59:34.822045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.821974 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4"} err="failed to get container status \"ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4\": rpc error: code = NotFound desc = could not find container \"ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4\": container with ID starting with ea068435784b60d0384ee420f14bf80d605925772efeaf1cfc83eb4f6ecb77a4 not found: ID does not exist" Oct 30 06:59:34.822045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.821998 5001 scope.go:122] "RemoveContainer" containerID="d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c" Oct 30 06:59:34.822269 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:34.822247 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c\": container with ID starting with d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c not found: ID does not exist" containerID="d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c" Oct 30 06:59:34.822334 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.822280 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c"} err="failed to get container status \"d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c\": rpc error: code = NotFound desc = could not find container \"d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c\": container with ID starting with d10a83903e2fb9cd1a65f01ef175b9503a0aac5d86cb5138ea705f5c1eb6a50c not found: ID does not exist" Oct 30 06:59:34.853869 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.853841 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g"] Oct 30 06:59:34.863180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:34.863159 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-bnv5g"] Oct 30 06:59:35.184702 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:35.184663 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" podUID="dd5eddac-dbe2-485e-8c7b-1fec89d02eeb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:36.117130 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:36.117096 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" path="/var/lib/kubelet/pods/f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78/volumes" Oct 30 06:59:37.470051 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:37.470009 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" podUID="8efc4083-009c-45f8-b535-7dff63289742" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:37.470598 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:37.470080 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" podUID="75d360de-4c0c-4d3f-bc06-1ea1922e9f43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:38.267216 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:38.267164 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 06:59:38.940042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:38.939974 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2"] Oct 30 06:59:38.967045 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:38.967012 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5"] Oct 30 06:59:39.135648 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:39.135613 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk"] Oct 30 06:59:39.139073 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:39.139043 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6e2548-stwlk"] Oct 30 06:59:39.662110 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:39.662080 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx"] Oct 30 06:59:39.690455 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:39.690425 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9"] Oct 30 06:59:39.844883 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:39.844854 5001 generic.go:356] "Generic (PLEG): container finished" podID="8efc4083-009c-45f8-b535-7dff63289742" containerID="8d2484a8a870f627e85ae4b37fb447e3ee44733afbcded16021dd4e8aa4f108c" exitCode=0 Oct 30 06:59:39.845016 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:39.844938 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" event={"ID":"8efc4083-009c-45f8-b535-7dff63289742","Type":"ContainerDied","Data":"8d2484a8a870f627e85ae4b37fb447e3ee44733afbcded16021dd4e8aa4f108c"} Oct 30 06:59:39.847180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:39.846839 5001 generic.go:356] "Generic (PLEG): container finished" podID="75d360de-4c0c-4d3f-bc06-1ea1922e9f43" containerID="2cd7ebec86e493d6bb27627fba5bc8185d9d9b30775015808af77ecb44386432" exitCode=0 Oct 30 06:59:39.847180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:39.847076 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" event={"ID":"75d360de-4c0c-4d3f-bc06-1ea1922e9f43","Type":"ContainerDied","Data":"2cd7ebec86e493d6bb27627fba5bc8185d9d9b30775015808af77ecb44386432"} Oct 30 06:59:39.980851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:39.980765 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:59:40.033396 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.033367 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:59:40.071837 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.071805 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-kube-api-access-chl8n\" (UniqueName: \"kubernetes.io/projected/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-kube-api-access-chl8n\") pod \"75d360de-4c0c-4d3f-bc06-1ea1922e9f43\" (UID: \"75d360de-4c0c-4d3f-bc06-1ea1922e9f43\") " Oct 30 06:59:40.072028 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.071888 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-plugins-config-volume\") pod \"75d360de-4c0c-4d3f-bc06-1ea1922e9f43\" (UID: \"75d360de-4c0c-4d3f-bc06-1ea1922e9f43\") " Oct 30 06:59:40.072354 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.072326 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-plugins-config-volume" pod "75d360de-4c0c-4d3f-bc06-1ea1922e9f43" (UID: "75d360de-4c0c-4d3f-bc06-1ea1922e9f43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:40.072647 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.072624 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:40.074781 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.074750 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-kube-api-access-chl8n" pod "75d360de-4c0c-4d3f-bc06-1ea1922e9f43" (UID: "75d360de-4c0c-4d3f-bc06-1ea1922e9f43"). InnerVolumeSpecName "kube-api-access-chl8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:40.121159 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.121121 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e76528-1ac2-4b14-b467-6b1bd75a3e68" path="/var/lib/kubelet/pods/60e76528-1ac2-4b14-b467-6b1bd75a3e68/volumes" Oct 30 06:59:40.173957 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.173928 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efc4083-009c-45f8-b535-7dff63289742-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efc4083-009c-45f8-b535-7dff63289742-plugins-config-volume\") pod \"8efc4083-009c-45f8-b535-7dff63289742\" (UID: \"8efc4083-009c-45f8-b535-7dff63289742\") " Oct 30 06:59:40.174126 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.174063 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efc4083-009c-45f8-b535-7dff63289742-kube-api-access-qcwrl\" (UniqueName: \"kubernetes.io/projected/8efc4083-009c-45f8-b535-7dff63289742-kube-api-access-qcwrl\") pod \"8efc4083-009c-45f8-b535-7dff63289742\" (UID: \"8efc4083-009c-45f8-b535-7dff63289742\") " Oct 30 06:59:40.174382 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.174359 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efc4083-009c-45f8-b535-7dff63289742-plugins-config-volume" pod "8efc4083-009c-45f8-b535-7dff63289742" (UID: "8efc4083-009c-45f8-b535-7dff63289742"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:40.174582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.174567 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chl8n\" (UniqueName: \"kubernetes.io/projected/75d360de-4c0c-4d3f-bc06-1ea1922e9f43-kube-api-access-chl8n\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:40.174634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.174585 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efc4083-009c-45f8-b535-7dff63289742-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:40.176319 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.176285 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efc4083-009c-45f8-b535-7dff63289742-kube-api-access-qcwrl" pod "8efc4083-009c-45f8-b535-7dff63289742" (UID: "8efc4083-009c-45f8-b535-7dff63289742"). InnerVolumeSpecName "kube-api-access-qcwrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:40.283390 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.279459 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcwrl\" (UniqueName: \"kubernetes.io/projected/8efc4083-009c-45f8-b535-7dff63289742-kube-api-access-qcwrl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:40.861865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.861834 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" Oct 30 06:59:40.861865 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.861858 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d" event={"ID":"8efc4083-009c-45f8-b535-7dff63289742","Type":"ContainerDied","Data":"a5bdae326110c79f1d72c03fa9de5c85db61ae84197858f95e3cd4fb957eec60"} Oct 30 06:59:40.862136 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.861948 5001 scope.go:122] "RemoveContainer" containerID="8d2484a8a870f627e85ae4b37fb447e3ee44733afbcded16021dd4e8aa4f108c" Oct 30 06:59:40.863633 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.863614 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" Oct 30 06:59:40.863766 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.863626 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8" event={"ID":"75d360de-4c0c-4d3f-bc06-1ea1922e9f43","Type":"ContainerDied","Data":"bd446deffffb073c0ea81b42f9184f7b394f49d94a0812e70d4b93d2a54c5a9f"} Oct 30 06:59:40.879121 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.879093 5001 scope.go:122] "RemoveContainer" containerID="2cd7ebec86e493d6bb27627fba5bc8185d9d9b30775015808af77ecb44386432" Oct 30 06:59:40.915977 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.915920 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d"] Oct 30 06:59:40.927477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.927456 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d8df2-5978475675-8pv7d"] Oct 30 06:59:40.967971 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.967761 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8"] Oct 30 06:59:40.986948 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:40.986801 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d8df2-668896bb8b-wg5j8"] Oct 30 06:59:42.117831 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:42.117798 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d360de-4c0c-4d3f-bc06-1ea1922e9f43" path="/var/lib/kubelet/pods/75d360de-4c0c-4d3f-bc06-1ea1922e9f43/volumes" Oct 30 06:59:42.118361 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:42.118337 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efc4083-009c-45f8-b535-7dff63289742" path="/var/lib/kubelet/pods/8efc4083-009c-45f8-b535-7dff63289742/volumes" Oct 30 06:59:43.186056 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:43.186025 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4"] Oct 30 06:59:43.198477 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:43.198444 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-645f2b-rswk4"] Oct 30 06:59:43.941151 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:43.941095 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" podUID="1131c14c-64e3-4840-88b7-50b0bde68434" containerName="payload-processing" containerID="cri-o://e4cf4464467e1b947aad230ab5e3cfa63fed9bfd34b6bc80eaf02b9c062ab25d" gracePeriod=25 Oct 30 06:59:43.967484 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:43.967422 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" podUID="5d03a7c2-0012-42dd-afd3-03c0d3c468e3" containerName="payload-pre-processing" containerID="cri-o://cb0dc7699158a55cbeb4dc11003206fa8bbdc59eae46e3c362d7f7d5d52caee1" gracePeriod=25 Oct 30 06:59:44.117362 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:44.117322 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679edb69-a667-4c6f-856e-4b62fb069d63" path="/var/lib/kubelet/pods/679edb69-a667-4c6f-856e-4b62fb069d63/volumes" Oct 30 06:59:44.662600 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:44.662559 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" podUID="e80eaafa-a522-400a-802f-0a54e846cc3d" containerName="payload-processing" containerID="cri-o://f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb" gracePeriod=25 Oct 30 06:59:44.691617 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:44.691579 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" podUID="6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4" containerName="payload-pre-processing" containerID="cri-o://21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e" gracePeriod=25 Oct 30 06:59:45.184830 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:45.184790 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" podUID="dd5eddac-dbe2-485e-8c7b-1fec89d02eeb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:45.624208 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:45.624162 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" podUID="1131c14c-64e3-4840-88b7-50b0bde68434" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:45.624494 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:45.624158 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" podUID="5d03a7c2-0012-42dd-afd3-03c0d3c468e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:46.743890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:46.743856 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n"] Oct 30 06:59:46.744476 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:46.744086 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" containerID="cri-o://8060131ed1d32b399e0275e31c63605774df70eff2fdb267cab81976a8a06e95" gracePeriod=30 Oct 30 06:59:47.356032 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.355990 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz"] Oct 30 06:59:47.356634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356612 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" Oct 30 06:59:47.356634 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356634 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356656 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8efc4083-009c-45f8-b535-7dff63289742" containerName="payload-pre-processing" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356663 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efc4083-009c-45f8-b535-7dff63289742" containerName="payload-pre-processing" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356681 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356688 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356720 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356729 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356749 5001 cpu_manager.go:414] "containerMap: removing container" podUID="679edb69-a667-4c6f-856e-4b62fb069d63" containerName="revoke-keys" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356755 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679edb69-a667-4c6f-856e-4b62fb069d63" containerName="revoke-keys" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356763 5001 cpu_manager.go:414] "containerMap: removing container" podUID="75d360de-4c0c-4d3f-bc06-1ea1922e9f43" containerName="payload-processing" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356770 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d360de-4c0c-4d3f-bc06-1ea1922e9f43" containerName="payload-processing" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356791 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f9a8053a-ef53-4211-af13-dd7cc881cc91" containerName="revoke-keys" Oct 30 06:59:47.356826 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356799 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a8053a-ef53-4211-af13-dd7cc881cc91" containerName="revoke-keys" Oct 30 06:59:47.357482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356955 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" Oct 30 06:59:47.357482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356970 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" Oct 30 06:59:47.357482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356981 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="679edb69-a667-4c6f-856e-4b62fb069d63" containerName="revoke-keys" Oct 30 06:59:47.357482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.356992 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d360de-4c0c-4d3f-bc06-1ea1922e9f43" containerName="payload-processing" Oct 30 06:59:47.357482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.357003 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a8053a-ef53-4211-af13-dd7cc881cc91" containerName="revoke-keys" Oct 30 06:59:47.357482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.357015 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" Oct 30 06:59:47.357482 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.357024 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efc4083-009c-45f8-b535-7dff63289742" containerName="payload-pre-processing" Oct 30 06:59:47.361930 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.361908 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:47.383694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.383664 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:47.383827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.383744 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:47.383827 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.383777 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q9ft\" (UniqueName: \"kubernetes.io/projected/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-kube-api-access-4q9ft\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:47.384679 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.384655 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz"] Oct 30 06:59:47.390851 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.390828 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:59:47.390966 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.390924 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:59:47.458493 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.458459 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:59:47.458644 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.458516 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:59:47.484553 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.484521 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:47.484698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.484607 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:47.484698 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.484637 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q9ft\" (UniqueName: \"kubernetes.io/projected/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-kube-api-access-4q9ft\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:47.485210 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.485189 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:47.487409 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.487386 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:47.508234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.508208 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q9ft\" (UniqueName: \"kubernetes.io/projected/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-kube-api-access-4q9ft\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:47.676811 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.676711 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:47.985880 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:47.985853 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz"] Oct 30 06:59:47.988839 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:59:47.988799 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edb5e8f_d923_48e9_b3f7_b1ade9d339e4.slice/crio-29138e702d53e47c668d7f6d8c0a67c710990ce0784d1cc0a212af8af72ba6a5 WatchSource:0}: Error finding container 29138e702d53e47c668d7f6d8c0a67c710990ce0784d1cc0a212af8af72ba6a5: Status 404 returned error can't find the container with id 29138e702d53e47c668d7f6d8c0a67c710990ce0784d1cc0a212af8af72ba6a5 Oct 30 06:59:48.646335 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:48.646181 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 06:59:48.948060 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:48.948025 5001 generic.go:356] "Generic (PLEG): container finished" podID="1edb5e8f-d923-48e9-b3f7-b1ade9d339e4" containerID="a9a9ab5479390a06cf26e073dc67d140346c1bfd615cd69d556dba343032ffb1" exitCode=0 Oct 30 06:59:48.948247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:48.948119 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" event={"ID":"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4","Type":"ContainerDied","Data":"a9a9ab5479390a06cf26e073dc67d140346c1bfd615cd69d556dba343032ffb1"} Oct 30 06:59:48.948247 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:48.948175 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" event={"ID":"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4","Type":"ContainerStarted","Data":"29138e702d53e47c668d7f6d8c0a67c710990ce0784d1cc0a212af8af72ba6a5"} Oct 30 06:59:49.447114 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.447051 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:59:49.509022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.508990 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-kube-api-access-pfpgk\" (UniqueName: \"kubernetes.io/projected/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-kube-api-access-pfpgk\") pod \"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb\" (UID: \"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb\") " Oct 30 06:59:49.509193 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.509105 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-plugins-config-volume\") pod \"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb\" (UID: \"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb\") " Oct 30 06:59:49.509630 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.509604 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-plugins-config-volume" pod "dd5eddac-dbe2-485e-8c7b-1fec89d02eeb" (UID: "dd5eddac-dbe2-485e-8c7b-1fec89d02eeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:49.511469 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.511443 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-kube-api-access-pfpgk" pod "dd5eddac-dbe2-485e-8c7b-1fec89d02eeb" (UID: "dd5eddac-dbe2-485e-8c7b-1fec89d02eeb"). InnerVolumeSpecName "kube-api-access-pfpgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:49.610529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.610495 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfpgk\" (UniqueName: \"kubernetes.io/projected/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-kube-api-access-pfpgk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:49.610529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.610531 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:49.957535 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.957493 5001 generic.go:356] "Generic (PLEG): container finished" podID="dd5eddac-dbe2-485e-8c7b-1fec89d02eeb" containerID="596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974" exitCode=0 Oct 30 06:59:49.957751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.957562 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" Oct 30 06:59:49.957751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.957579 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" event={"ID":"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb","Type":"ContainerDied","Data":"596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974"} Oct 30 06:59:49.957751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.957670 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m" event={"ID":"dd5eddac-dbe2-485e-8c7b-1fec89d02eeb","Type":"ContainerDied","Data":"7d45fc6a46af55efb72b51973c4b2e79fed25de4824df0eadf4aed84397df343"} Oct 30 06:59:49.957751 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.957700 5001 scope.go:122] "RemoveContainer" containerID="596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974" Oct 30 06:59:49.970736 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.970713 5001 scope.go:122] "RemoveContainer" containerID="596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974" Oct 30 06:59:49.971338 ip-10-0-128-31 kubenswrapper[5001]: E1030 06:59:49.971026 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974\": container with ID starting with 596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974 not found: ID does not exist" containerID="596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974" Oct 30 06:59:49.971338 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:49.971060 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974"} err="failed to get container status \"596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974\": rpc error: code = NotFound desc = could not find container \"596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974\": container with ID starting with 596b3b3fb68c47a9b1c7c04e267599fc1cd8223e583840bfa33f4dc1a5e9f974 not found: ID does not exist" Oct 30 06:59:50.009451 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.009418 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m"] Oct 30 06:59:50.012868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.012842 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6e2548-6647c5c56-8km4m"] Oct 30 06:59:50.118059 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.118027 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd5eddac-dbe2-485e-8c7b-1fec89d02eeb" path="/var/lib/kubelet/pods/dd5eddac-dbe2-485e-8c7b-1fec89d02eeb/volumes" Oct 30 06:59:50.127907 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.127879 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:50.215823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.215726 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-xks-ca\") pod \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " Oct 30 06:59:50.215823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.215771 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-openshift-ca\") pod \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " Oct 30 06:59:50.215823 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.215819 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-kube-api-access-4q9ft\" (UniqueName: \"kubernetes.io/projected/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-kube-api-access-4q9ft\") pod \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\" (UID: \"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4\") " Oct 30 06:59:50.216180 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.216156 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-openshift-ca" pod "1edb5e8f-d923-48e9-b3f7-b1ade9d339e4" (UID: "1edb5e8f-d923-48e9-b3f7-b1ade9d339e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:50.216299 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.216280 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:50.218135 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.218105 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-xks-ca" pod "1edb5e8f-d923-48e9-b3f7-b1ade9d339e4" (UID: "1edb5e8f-d923-48e9-b3f7-b1ade9d339e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:50.218246 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.218154 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-kube-api-access-4q9ft" pod "1edb5e8f-d923-48e9-b3f7-b1ade9d339e4" (UID: "1edb5e8f-d923-48e9-b3f7-b1ade9d339e4"). InnerVolumeSpecName "kube-api-access-4q9ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:50.317783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.317750 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q9ft\" (UniqueName: \"kubernetes.io/projected/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-kube-api-access-4q9ft\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:50.317783 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.317779 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:50.907149 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.907115 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd"] Oct 30 06:59:50.907688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.907361 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" containerID="cri-o://7ffb1aa6bebd50764ae4cd7b5356eebb96c9adbe904eb907973c5e18eb6a60b1" gracePeriod=30 Oct 30 06:59:50.945419 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.945384 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" podUID="6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:50.945578 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.945395 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" podUID="e80eaafa-a522-400a-802f-0a54e846cc3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:50.970385 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.970359 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" Oct 30 06:59:50.970589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.970361 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz" event={"ID":"1edb5e8f-d923-48e9-b3f7-b1ade9d339e4","Type":"ContainerDied","Data":"29138e702d53e47c668d7f6d8c0a67c710990ce0784d1cc0a212af8af72ba6a5"} Oct 30 06:59:50.970589 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:50.970491 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29138e702d53e47c668d7f6d8c0a67c710990ce0784d1cc0a212af8af72ba6a5" Oct 30 06:59:51.515785 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.515744 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5"] Oct 30 06:59:51.516401 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.516380 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1edb5e8f-d923-48e9-b3f7-b1ade9d339e4" containerName="revoke-keys" Oct 30 06:59:51.516529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.516402 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edb5e8f-d923-48e9-b3f7-b1ade9d339e4" containerName="revoke-keys" Oct 30 06:59:51.516529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.516422 5001 cpu_manager.go:414] "containerMap: removing container" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" Oct 30 06:59:51.516529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.516429 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9fe738-2674-4d41-a255-1c2a057d4ca5" containerName="istio-proxy" Oct 30 06:59:51.516529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.516455 5001 cpu_manager.go:414] "containerMap: removing container" podUID="dd5eddac-dbe2-485e-8c7b-1fec89d02eeb" containerName="payload-processing" Oct 30 06:59:51.516529 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.516462 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5eddac-dbe2-485e-8c7b-1fec89d02eeb" containerName="payload-processing" Oct 30 06:59:51.516868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.516630 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edb5e8f-d923-48e9-b3f7-b1ade9d339e4" containerName="revoke-keys" Oct 30 06:59:51.516868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.516650 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5eddac-dbe2-485e-8c7b-1fec89d02eeb" containerName="payload-processing" Oct 30 06:59:51.516868 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.516664 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bccfa7-5e6e-4e3e-8139-2c2ff8c2da78" containerName="istio-proxy" Oct 30 06:59:51.520234 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.520211 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:51.535919 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.535882 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:59:51.537421 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.536460 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:59:51.540872 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.540844 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5"] Oct 30 06:59:51.601398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.601356 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:59:51.601398 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.601385 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:59:51.634093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.634057 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:51.634291 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.634274 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p95df\" (UniqueName: \"kubernetes.io/projected/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-kube-api-access-p95df\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:51.634406 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.634328 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:51.735068 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.735033 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p95df\" (UniqueName: \"kubernetes.io/projected/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-kube-api-access-p95df\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:51.735267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.735075 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:51.735267 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.735116 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:51.735806 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.735775 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:51.737713 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.737688 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:51.758694 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.758669 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p95df\" (UniqueName: \"kubernetes.io/projected/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-kube-api-access-p95df\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:51.834731 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.834704 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:51.998410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.997965 5001 generic.go:356] "Generic (PLEG): container finished" podID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerID="8060131ed1d32b399e0275e31c63605774df70eff2fdb267cab81976a8a06e95" exitCode=0 Oct 30 06:59:51.998410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.998233 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" event={"ID":"1dbe1898-1454-48fe-8e9f-c1bcf0600b44","Type":"ContainerDied","Data":"8060131ed1d32b399e0275e31c63605774df70eff2fdb267cab81976a8a06e95"} Oct 30 06:59:51.998410 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:51.998298 5001 scope.go:122] "RemoveContainer" containerID="1be51816e7cd52820608b7fc1dc43ca669c6c749a5aedb89cf3d25dcdaf20018" Oct 30 06:59:52.044894 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.044871 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:59:52.139345 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139315 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-certs\") pod \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " Oct 30 06:59:52.139502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139361 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istiod-ca-cert\") pod \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " Oct 30 06:59:52.139502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139386 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-kube-api-access-29l8z\" (UniqueName: \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-kube-api-access-29l8z\") pod \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " Oct 30 06:59:52.139502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139417 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-token\" (UniqueName: \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-token\") pod \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " Oct 30 06:59:52.139502 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139454 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-socket\") pod \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " Oct 30 06:59:52.139740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139589 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-podinfo\") pod \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " Oct 30 06:59:52.139740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139637 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-data\") pod \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " Oct 30 06:59:52.139740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139595 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-certs" pod "1dbe1898-1454-48fe-8e9f-c1bcf0600b44" (UID: "1dbe1898-1454-48fe-8e9f-c1bcf0600b44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:52.139740 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139728 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-credential-socket\") pod \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " Oct 30 06:59:52.139985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139776 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-envoy\") pod \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\" (UID: \"1dbe1898-1454-48fe-8e9f-c1bcf0600b44\") " Oct 30 06:59:52.139985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139812 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-socket" pod "1dbe1898-1454-48fe-8e9f-c1bcf0600b44" (UID: "1dbe1898-1454-48fe-8e9f-c1bcf0600b44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:52.139985 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139895 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istiod-ca-cert" pod "1dbe1898-1454-48fe-8e9f-c1bcf0600b44" (UID: "1dbe1898-1454-48fe-8e9f-c1bcf0600b44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:52.140162 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.139983 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-credential-socket" pod "1dbe1898-1454-48fe-8e9f-c1bcf0600b44" (UID: "1dbe1898-1454-48fe-8e9f-c1bcf0600b44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:52.140252 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.140214 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-data" pod "1dbe1898-1454-48fe-8e9f-c1bcf0600b44" (UID: "1dbe1898-1454-48fe-8e9f-c1bcf0600b44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:52.140856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.140838 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:52.140856 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.140856 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:52.141042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.140867 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:52.141042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.140875 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:52.141042 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.140883 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:52.142292 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.142240 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-token" pod "1dbe1898-1454-48fe-8e9f-c1bcf0600b44" (UID: "1dbe1898-1454-48fe-8e9f-c1bcf0600b44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:52.142507 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.142488 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-kube-api-access-29l8z" pod "1dbe1898-1454-48fe-8e9f-c1bcf0600b44" (UID: "1dbe1898-1454-48fe-8e9f-c1bcf0600b44"). InnerVolumeSpecName "kube-api-access-29l8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:52.142700 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.142684 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-podinfo" pod "1dbe1898-1454-48fe-8e9f-c1bcf0600b44" (UID: "1dbe1898-1454-48fe-8e9f-c1bcf0600b44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:59:52.142861 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.142846 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-envoy" pod "1dbe1898-1454-48fe-8e9f-c1bcf0600b44" (UID: "1dbe1898-1454-48fe-8e9f-c1bcf0600b44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:52.154263 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.154241 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5"] Oct 30 06:59:52.157609 ip-10-0-128-31 kubenswrapper[5001]: W1030 06:59:52.157587 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ee0f28_85c1_41ad_9d4b_03e216bd641e.slice/crio-afdf250ba8d91c76a6b57cb5150f603638c02d575162add462c40a7f85239032 WatchSource:0}: Error finding container afdf250ba8d91c76a6b57cb5150f603638c02d575162add462c40a7f85239032: Status 404 returned error can't find the container with id afdf250ba8d91c76a6b57cb5150f603638c02d575162add462c40a7f85239032 Oct 30 06:59:52.241518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.241491 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:52.241518 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.241516 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:52.241688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.241526 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:52.241688 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:52.241536 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29l8z\" (UniqueName: \"kubernetes.io/projected/1dbe1898-1454-48fe-8e9f-c1bcf0600b44-kube-api-access-29l8z\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:53.020298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:53.020206 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" Oct 30 06:59:53.020298 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:53.020239 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n" event={"ID":"1dbe1898-1454-48fe-8e9f-c1bcf0600b44","Type":"ContainerDied","Data":"df0f10736befcf3248e906c6beff8f42cc231df78eaadc4917937328f93c3c5d"} Oct 30 06:59:53.020890 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:53.020375 5001 scope.go:122] "RemoveContainer" containerID="8060131ed1d32b399e0275e31c63605774df70eff2fdb267cab81976a8a06e95" Oct 30 06:59:53.022288 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:53.022258 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" event={"ID":"f0ee0f28-85c1-41ad-9d4b-03e216bd641e","Type":"ContainerStarted","Data":"711fa0411b3971ea8aaecb1866a43273eff6344692e2d71ed13fb803485aef24"} Oct 30 06:59:53.022450 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:53.022327 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" event={"ID":"f0ee0f28-85c1-41ad-9d4b-03e216bd641e","Type":"ContainerStarted","Data":"afdf250ba8d91c76a6b57cb5150f603638c02d575162add462c40a7f85239032"} Oct 30 06:59:53.074185 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:53.074142 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n"] Oct 30 06:59:53.083566 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:53.083539 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6e2548-openshift-default-86665b47c5-q785n"] Oct 30 06:59:53.114022 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:53.113977 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" podStartSLOduration=2.11396396 podStartE2EDuration="2.11396396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:59:53.112565258 +0000 UTC m=+4007.108522269" watchObservedRunningTime="2026-10-30 06:59:53.11396396 +0000 UTC m=+4007.109920970" Oct 30 06:59:54.033144 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:54.033109 5001 generic.go:356] "Generic (PLEG): container finished" podID="f0ee0f28-85c1-41ad-9d4b-03e216bd641e" containerID="711fa0411b3971ea8aaecb1866a43273eff6344692e2d71ed13fb803485aef24" exitCode=0 Oct 30 06:59:54.033632 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:54.033197 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" event={"ID":"f0ee0f28-85c1-41ad-9d4b-03e216bd641e","Type":"ContainerDied","Data":"711fa0411b3971ea8aaecb1866a43273eff6344692e2d71ed13fb803485aef24"} Oct 30 06:59:54.117701 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:54.117671 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" path="/var/lib/kubelet/pods/1dbe1898-1454-48fe-8e9f-c1bcf0600b44/volumes" Oct 30 06:59:54.726034 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:54.726001 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8"] Oct 30 06:59:54.747355 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:54.743698 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-adcdcb-ltnp8"] Oct 30 06:59:55.189499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.189474 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:55.273499 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.273467 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-kube-api-access-p95df\" (UniqueName: \"kubernetes.io/projected/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-kube-api-access-p95df\") pod \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " Oct 30 06:59:55.273706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.273510 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-openshift-ca\") pod \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " Oct 30 06:59:55.273706 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.273614 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-xks-ca\") pod \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\" (UID: \"f0ee0f28-85c1-41ad-9d4b-03e216bd641e\") " Oct 30 06:59:55.273999 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.273969 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-openshift-ca" pod "f0ee0f28-85c1-41ad-9d4b-03e216bd641e" (UID: "f0ee0f28-85c1-41ad-9d4b-03e216bd641e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:55.275748 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.275727 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-xks-ca" pod "f0ee0f28-85c1-41ad-9d4b-03e216bd641e" (UID: "f0ee0f28-85c1-41ad-9d4b-03e216bd641e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:55.275990 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.275971 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-kube-api-access-p95df" pod "f0ee0f28-85c1-41ad-9d4b-03e216bd641e" (UID: "f0ee0f28-85c1-41ad-9d4b-03e216bd641e"). InnerVolumeSpecName "kube-api-access-p95df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:55.374551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.374520 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p95df\" (UniqueName: \"kubernetes.io/projected/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-kube-api-access-p95df\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:55.374551 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.374552 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:55.374754 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.374566 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ee0f28-85c1-41ad-9d4b-03e216bd641e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:55.624459 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.624421 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" podUID="1131c14c-64e3-4840-88b7-50b0bde68434" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:55.625430 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:55.625352 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" podUID="5d03a7c2-0012-42dd-afd3-03c0d3c468e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:56.053914 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.053883 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" Oct 30 06:59:56.054093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.053888 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5" event={"ID":"f0ee0f28-85c1-41ad-9d4b-03e216bd641e","Type":"ContainerDied","Data":"afdf250ba8d91c76a6b57cb5150f603638c02d575162add462c40a7f85239032"} Oct 30 06:59:56.054093 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.054047 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afdf250ba8d91c76a6b57cb5150f603638c02d575162add462c40a7f85239032" Oct 30 06:59:56.056320 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.056284 5001 generic.go:356] "Generic (PLEG): container finished" podID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerID="7ffb1aa6bebd50764ae4cd7b5356eebb96c9adbe904eb907973c5e18eb6a60b1" exitCode=0 Oct 30 06:59:56.056454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.056351 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" event={"ID":"c7f2a909-e267-4a73-89a1-37170fa5bc73","Type":"ContainerDied","Data":"7ffb1aa6bebd50764ae4cd7b5356eebb96c9adbe904eb907973c5e18eb6a60b1"} Oct 30 06:59:56.056454 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.056402 5001 scope.go:122] "RemoveContainer" containerID="e88bddbf3606f0ae1de8c3391244724846231af8060b886a17b2e2e54189ec41" Oct 30 06:59:56.119350 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.119286 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a8053a-ef53-4211-af13-dd7cc881cc91" path="/var/lib/kubelet/pods/f9a8053a-ef53-4211-af13-dd7cc881cc91/volumes" Oct 30 06:59:56.165571 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.165545 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:59:56.285564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.285475 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-podinfo\") pod \"c7f2a909-e267-4a73-89a1-37170fa5bc73\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " Oct 30 06:59:56.285564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.285532 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-envoy\") pod \"c7f2a909-e267-4a73-89a1-37170fa5bc73\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " Oct 30 06:59:56.285564 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.285555 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-certs\") pod \"c7f2a909-e267-4a73-89a1-37170fa5bc73\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " Oct 30 06:59:56.286125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.285581 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f2a909-e267-4a73-89a1-37170fa5bc73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2a909-e267-4a73-89a1-37170fa5bc73-istiod-ca-cert\") pod \"c7f2a909-e267-4a73-89a1-37170fa5bc73\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " Oct 30 06:59:56.286125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.285609 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-credential-socket\") pod \"c7f2a909-e267-4a73-89a1-37170fa5bc73\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " Oct 30 06:59:56.286125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.285697 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-socket\") pod \"c7f2a909-e267-4a73-89a1-37170fa5bc73\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " Oct 30 06:59:56.286125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.285796 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-token\") pod \"c7f2a909-e267-4a73-89a1-37170fa5bc73\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " Oct 30 06:59:56.286125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.285851 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-kube-api-access-ltxfk\" (UniqueName: \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-kube-api-access-ltxfk\") pod \"c7f2a909-e267-4a73-89a1-37170fa5bc73\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " Oct 30 06:59:56.286125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.285897 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-data\") pod \"c7f2a909-e267-4a73-89a1-37170fa5bc73\" (UID: \"c7f2a909-e267-4a73-89a1-37170fa5bc73\") " Oct 30 06:59:56.286125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.285905 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-certs" pod "c7f2a909-e267-4a73-89a1-37170fa5bc73" (UID: "c7f2a909-e267-4a73-89a1-37170fa5bc73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:56.286125 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.286067 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f2a909-e267-4a73-89a1-37170fa5bc73-istiod-ca-cert" pod "c7f2a909-e267-4a73-89a1-37170fa5bc73" (UID: "c7f2a909-e267-4a73-89a1-37170fa5bc73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:56.286807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.286287 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-data" pod "c7f2a909-e267-4a73-89a1-37170fa5bc73" (UID: "c7f2a909-e267-4a73-89a1-37170fa5bc73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:56.286807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.286441 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:56.286807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.286457 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2a909-e267-4a73-89a1-37170fa5bc73-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:56.286807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.286466 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:56.286807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.286587 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-credential-socket" pod "c7f2a909-e267-4a73-89a1-37170fa5bc73" (UID: "c7f2a909-e267-4a73-89a1-37170fa5bc73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:56.286807 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.286630 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-socket" pod "c7f2a909-e267-4a73-89a1-37170fa5bc73" (UID: "c7f2a909-e267-4a73-89a1-37170fa5bc73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:56.288120 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.288090 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-envoy" pod "c7f2a909-e267-4a73-89a1-37170fa5bc73" (UID: "c7f2a909-e267-4a73-89a1-37170fa5bc73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:56.288277 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.288264 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-token" pod "c7f2a909-e267-4a73-89a1-37170fa5bc73" (UID: "c7f2a909-e267-4a73-89a1-37170fa5bc73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:56.288582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.288557 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-kube-api-access-ltxfk" pod "c7f2a909-e267-4a73-89a1-37170fa5bc73" (UID: "c7f2a909-e267-4a73-89a1-37170fa5bc73"). InnerVolumeSpecName "kube-api-access-ltxfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:56.288582 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.288564 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-podinfo" pod "c7f2a909-e267-4a73-89a1-37170fa5bc73" (UID: "c7f2a909-e267-4a73-89a1-37170fa5bc73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:59:56.387373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.387335 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltxfk\" (UniqueName: \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-kube-api-access-ltxfk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:56.387373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.387370 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:56.387373 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.387379 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:56.387614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.387388 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:56.387614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.387398 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2a909-e267-4a73-89a1-37170fa5bc73-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:56.387614 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:56.387407 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2a909-e267-4a73-89a1-37170fa5bc73-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 06:59:57.067417 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:57.067384 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" Oct 30 06:59:57.067618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:57.067383 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd" event={"ID":"c7f2a909-e267-4a73-89a1-37170fa5bc73","Type":"ContainerDied","Data":"c4641f16543f9924b2b667ed3cef5c03654c946ad832128f836f86bc78047a69"} Oct 30 06:59:57.067618 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:57.067543 5001 scope.go:122] "RemoveContainer" containerID="7ffb1aa6bebd50764ae4cd7b5356eebb96c9adbe904eb907973c5e18eb6a60b1" Oct 30 06:59:57.119154 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:57.119126 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd"] Oct 30 06:59:57.131640 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:57.131610 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-645f2b-openshift-default-78db55bf75-msmtd"] Oct 30 06:59:58.116352 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:58.116285 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" path="/var/lib/kubelet/pods/c7f2a909-e267-4a73-89a1-37170fa5bc73/volumes" Oct 30 06:59:59.367433 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:59.367395 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz"] Oct 30 06:59:59.376909 ip-10-0-128-31 kubenswrapper[5001]: I1030 06:59:59.376877 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-868b1f-rcpdz"] Oct 30 07:00:00.116759 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.116723 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edb5e8f-d923-48e9-b3f7-b1ade9d339e4" path="/var/lib/kubelet/pods/1edb5e8f-d923-48e9-b3f7-b1ade9d339e4/volumes" Oct 30 07:00:00.149213 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149179 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc"] Oct 30 07:00:00.149755 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149740 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" Oct 30 07:00:00.149817 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149756 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" Oct 30 07:00:00.149817 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149778 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" Oct 30 07:00:00.149817 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149784 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" Oct 30 07:00:00.149817 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149809 5001 cpu_manager.go:414] "containerMap: removing container" podUID="f0ee0f28-85c1-41ad-9d4b-03e216bd641e" containerName="revoke-keys" Oct 30 07:00:00.149817 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149813 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ee0f28-85c1-41ad-9d4b-03e216bd641e" containerName="revoke-keys" Oct 30 07:00:00.149998 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149825 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" Oct 30 07:00:00.149998 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149829 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" Oct 30 07:00:00.149998 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149912 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" Oct 30 07:00:00.149998 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149924 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" Oct 30 07:00:00.149998 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.149932 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ee0f28-85c1-41ad-9d4b-03e216bd641e" containerName="revoke-keys" Oct 30 07:00:00.155188 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.155140 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" Oct 30 07:00:00.156072 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.156035 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh"] Oct 30 07:00:00.156920 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.156898 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" Oct 30 07:00:00.157001 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.156926 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" Oct 30 07:00:00.157097 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.157084 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbe1898-1454-48fe-8e9f-c1bcf0600b44" containerName="istio-proxy" Oct 30 07:00:00.157137 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.157110 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f2a909-e267-4a73-89a1-37170fa5bc73" containerName="istio-proxy" Oct 30 07:00:00.160855 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.160834 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr"] Oct 30 07:00:00.160990 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.160962 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" Oct 30 07:00:00.166166 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.166138 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" Oct 30 07:00:00.168557 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.168529 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml"] Oct 30 07:00:00.173712 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.172871 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" Oct 30 07:00:00.175731 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.174840 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:00:00.177663 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.177127 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:00:00.178666 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.178515 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh"] Oct 30 07:00:00.183641 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.183615 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 07:00:00.186416 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.186358 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc"] Oct 30 07:00:00.190609 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.190585 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr"] Oct 30 07:00:00.199791 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.199716 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml"] Oct 30 07:00:00.229863 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.229828 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkq8\" (UniqueName: \"kubernetes.io/projected/e45c44da-7337-4c2e-8a79-f6e76e853db6-kube-api-access-dzkq8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr\" (UID: \"e45c44da-7337-4c2e-8a79-f6e76e853db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" Oct 30 07:00:00.230057 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.230016 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc\" (UID: \"e8a8e120-452d-4dbc-8af4-09df0ce99e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" Oct 30 07:00:00.230139 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.230099 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh\" (UID: \"56d1c8a0-de78-4cc5-85e5-f538ce6237ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" Oct 30 07:00:00.230210 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.230139 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v48mb\" (UniqueName: \"kubernetes.io/projected/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-kube-api-access-v48mb\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh\" (UID: \"56d1c8a0-de78-4cc5-85e5-f538ce6237ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" Oct 30 07:00:00.230210 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.230171 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrcsj\" (UniqueName: \"kubernetes.io/projected/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-kube-api-access-xrcsj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc\" (UID: \"e8a8e120-452d-4dbc-8af4-09df0ce99e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" Oct 30 07:00:00.230346 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.230227 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df149d0-8b5f-4273-aad4-11a641230f8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml\" (UID: \"0df149d0-8b5f-4273-aad4-11a641230f8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" Oct 30 07:00:00.230346 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.230262 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45c44da-7337-4c2e-8a79-f6e76e853db6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr\" (UID: \"e45c44da-7337-4c2e-8a79-f6e76e853db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" Oct 30 07:00:00.230346 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.230291 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrj46\" (UniqueName: \"kubernetes.io/projected/0df149d0-8b5f-4273-aad4-11a641230f8d-kube-api-access-zrj46\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml\" (UID: \"0df149d0-8b5f-4273-aad4-11a641230f8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" Oct 30 07:00:00.331572 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.331533 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh\" (UID: \"56d1c8a0-de78-4cc5-85e5-f538ce6237ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" Oct 30 07:00:00.331572 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.331579 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v48mb\" (UniqueName: \"kubernetes.io/projected/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-kube-api-access-v48mb\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh\" (UID: \"56d1c8a0-de78-4cc5-85e5-f538ce6237ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" Oct 30 07:00:00.331813 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.331702 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrcsj\" (UniqueName: \"kubernetes.io/projected/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-kube-api-access-xrcsj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc\" (UID: \"e8a8e120-452d-4dbc-8af4-09df0ce99e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" Oct 30 07:00:00.331813 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.331776 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df149d0-8b5f-4273-aad4-11a641230f8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml\" (UID: \"0df149d0-8b5f-4273-aad4-11a641230f8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" Oct 30 07:00:00.331927 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.331818 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45c44da-7337-4c2e-8a79-f6e76e853db6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr\" (UID: \"e45c44da-7337-4c2e-8a79-f6e76e853db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" Oct 30 07:00:00.331927 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.331849 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrj46\" (UniqueName: \"kubernetes.io/projected/0df149d0-8b5f-4273-aad4-11a641230f8d-kube-api-access-zrj46\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml\" (UID: \"0df149d0-8b5f-4273-aad4-11a641230f8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" Oct 30 07:00:00.332372 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.332010 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkq8\" (UniqueName: \"kubernetes.io/projected/e45c44da-7337-4c2e-8a79-f6e76e853db6-kube-api-access-dzkq8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr\" (UID: \"e45c44da-7337-4c2e-8a79-f6e76e853db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" Oct 30 07:00:00.332372 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.332055 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc\" (UID: \"e8a8e120-452d-4dbc-8af4-09df0ce99e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" Oct 30 07:00:00.332461 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.332422 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh\" (UID: \"56d1c8a0-de78-4cc5-85e5-f538ce6237ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" Oct 30 07:00:00.332647 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.332626 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45c44da-7337-4c2e-8a79-f6e76e853db6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr\" (UID: \"e45c44da-7337-4c2e-8a79-f6e76e853db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" Oct 30 07:00:00.332687 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.332626 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df149d0-8b5f-4273-aad4-11a641230f8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml\" (UID: \"0df149d0-8b5f-4273-aad4-11a641230f8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" Oct 30 07:00:00.332728 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.332711 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc\" (UID: \"e8a8e120-452d-4dbc-8af4-09df0ce99e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" Oct 30 07:00:00.355720 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.355689 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkq8\" (UniqueName: \"kubernetes.io/projected/e45c44da-7337-4c2e-8a79-f6e76e853db6-kube-api-access-dzkq8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr\" (UID: \"e45c44da-7337-4c2e-8a79-f6e76e853db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" Oct 30 07:00:00.355900 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.355806 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrj46\" (UniqueName: \"kubernetes.io/projected/0df149d0-8b5f-4273-aad4-11a641230f8d-kube-api-access-zrj46\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml\" (UID: \"0df149d0-8b5f-4273-aad4-11a641230f8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" Oct 30 07:00:00.355973 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.355910 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v48mb\" (UniqueName: \"kubernetes.io/projected/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-kube-api-access-v48mb\") pod \"maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh\" (UID: \"56d1c8a0-de78-4cc5-85e5-f538ce6237ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" Oct 30 07:00:00.355973 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.355910 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrcsj\" (UniqueName: \"kubernetes.io/projected/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-kube-api-access-xrcsj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc\" (UID: \"e8a8e120-452d-4dbc-8af4-09df0ce99e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" Oct 30 07:00:00.482810 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.482700 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" Oct 30 07:00:00.493188 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.493149 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" Oct 30 07:00:00.506079 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.506045 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" Oct 30 07:00:00.517442 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.517413 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" Oct 30 07:00:00.814678 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.811950 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc"] Oct 30 07:00:00.847277 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.847243 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 07:00:00.873706 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.873665 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr"] Oct 30 07:00:00.879939 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.879895 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh"] Oct 30 07:00:00.889797 ip-10-0-128-31 kubenswrapper[5001]: W1030 07:00:00.889752 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d1c8a0_de78_4cc5_85e5_f538ce6237ba.slice/crio-f36ab47524129d7e5d595dd03843f1888605bc6f66a6411109a4ce9ee3062ccd WatchSource:0}: Error finding container f36ab47524129d7e5d595dd03843f1888605bc6f66a6411109a4ce9ee3062ccd: Status 404 returned error can't find the container with id f36ab47524129d7e5d595dd03843f1888605bc6f66a6411109a4ce9ee3062ccd Oct 30 07:00:00.918803 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.918779 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml"] Oct 30 07:00:00.920952 ip-10-0-128-31 kubenswrapper[5001]: W1030 07:00:00.920928 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df149d0_8b5f_4273_aad4_11a641230f8d.slice/crio-65bdbaa0c77845f3cb2d42a15b69ba95b015bbdac92f4a7f2571f2016ca75be4 WatchSource:0}: Error finding container 65bdbaa0c77845f3cb2d42a15b69ba95b015bbdac92f4a7f2571f2016ca75be4: Status 404 returned error can't find the container with id 65bdbaa0c77845f3cb2d42a15b69ba95b015bbdac92f4a7f2571f2016ca75be4 Oct 30 07:00:00.944584 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.944549 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" podUID="6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:00.944696 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:00.944587 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" podUID="e80eaafa-a522-400a-802f-0a54e846cc3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:01.109471 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:01.109428 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" event={"ID":"0df149d0-8b5f-4273-aad4-11a641230f8d","Type":"ContainerStarted","Data":"65bdbaa0c77845f3cb2d42a15b69ba95b015bbdac92f4a7f2571f2016ca75be4"} Oct 30 07:00:01.110838 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:01.110811 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" event={"ID":"e8a8e120-452d-4dbc-8af4-09df0ce99e6d","Type":"ContainerStarted","Data":"a68fe7f85461b9b51b24af6dfcc0152b8556144d339430b064c6346acb59acb0"} Oct 30 07:00:01.112246 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:01.112220 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" event={"ID":"56d1c8a0-de78-4cc5-85e5-f538ce6237ba","Type":"ContainerStarted","Data":"f36ab47524129d7e5d595dd03843f1888605bc6f66a6411109a4ce9ee3062ccd"} Oct 30 07:00:01.113513 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:01.113491 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" event={"ID":"e45c44da-7337-4c2e-8a79-f6e76e853db6","Type":"ContainerStarted","Data":"0743a90d79886f3f397e7d479a9b7447eec427bc5d021cd9d47d4b27a24367de"} Oct 30 07:00:02.025360 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.025319 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw"] Oct 30 07:00:02.139426 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.139388 5001 generic.go:356] "Generic (PLEG): container finished" podID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerID="f09531459141aacd4a30beac63406e2211cbca7edb86b91f88d8e8337dac6b9d" exitCode=60 Oct 30 07:00:02.140184 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.139806 5001 scope.go:122] "RemoveContainer" containerID="f09531459141aacd4a30beac63406e2211cbca7edb86b91f88d8e8337dac6b9d" Oct 30 07:00:02.140184 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.139814 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" event={"ID":"0df149d0-8b5f-4273-aad4-11a641230f8d","Type":"ContainerDied","Data":"f09531459141aacd4a30beac63406e2211cbca7edb86b91f88d8e8337dac6b9d"} Oct 30 07:00:02.142607 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.142582 5001 generic.go:356] "Generic (PLEG): container finished" podID="e8a8e120-452d-4dbc-8af4-09df0ce99e6d" containerID="40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2" exitCode=60 Oct 30 07:00:02.143060 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.143041 5001 scope.go:122] "RemoveContainer" containerID="40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2" Oct 30 07:00:02.143175 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.143122 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" event={"ID":"e8a8e120-452d-4dbc-8af4-09df0ce99e6d","Type":"ContainerDied","Data":"40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2"} Oct 30 07:00:02.147122 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.147097 5001 generic.go:356] "Generic (PLEG): container finished" podID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerID="be81e5c687abafddda1148b8c1f15356a60ff11d7b4e93f4c1d02d3f0e22ef62" exitCode=60 Oct 30 07:00:02.147345 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.147204 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" event={"ID":"56d1c8a0-de78-4cc5-85e5-f538ce6237ba","Type":"ContainerDied","Data":"be81e5c687abafddda1148b8c1f15356a60ff11d7b4e93f4c1d02d3f0e22ef62"} Oct 30 07:00:02.147471 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.147453 5001 scope.go:122] "RemoveContainer" containerID="be81e5c687abafddda1148b8c1f15356a60ff11d7b4e93f4c1d02d3f0e22ef62" Oct 30 07:00:02.172542 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.172507 5001 generic.go:356] "Generic (PLEG): container finished" podID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerID="ae9eed2e65e60fc40b39f8d159ef9d3166e0f02548f71d1e24d76b7126ce6a60" exitCode=60 Oct 30 07:00:02.172754 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.172624 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" event={"ID":"e45c44da-7337-4c2e-8a79-f6e76e853db6","Type":"ContainerDied","Data":"ae9eed2e65e60fc40b39f8d159ef9d3166e0f02548f71d1e24d76b7126ce6a60"} Oct 30 07:00:02.172944 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.172924 5001 scope.go:122] "RemoveContainer" containerID="ae9eed2e65e60fc40b39f8d159ef9d3166e0f02548f71d1e24d76b7126ce6a60" Oct 30 07:00:02.220489 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.220461 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc"] Oct 30 07:00:02.740616 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.739736 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26"] Oct 30 07:00:02.740616 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:02.739950 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" containerID="cri-o://a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada" gracePeriod=30 Oct 30 07:00:03.185205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:03.185159 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" event={"ID":"0df149d0-8b5f-4273-aad4-11a641230f8d","Type":"ContainerStarted","Data":"bc6ddda41f611f653185b2753f66b013faeea2d5545137c848de14e14a6c1004"} Oct 30 07:00:03.187857 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:03.187828 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" event={"ID":"e8a8e120-452d-4dbc-8af4-09df0ce99e6d","Type":"ContainerStarted","Data":"2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d"} Oct 30 07:00:03.187995 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:03.187874 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" podUID="e8a8e120-452d-4dbc-8af4-09df0ce99e6d" containerName="cleanup" containerID="cri-o://2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d" gracePeriod=30 Oct 30 07:00:03.190883 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:03.190853 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" event={"ID":"56d1c8a0-de78-4cc5-85e5-f538ce6237ba","Type":"ContainerStarted","Data":"fb62a3893ae45e1346459b4ccd4c7a40160a4fe871775230b0cca226f4ff39e5"} Oct 30 07:00:03.193331 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:03.193283 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" event={"ID":"e45c44da-7337-4c2e-8a79-f6e76e853db6","Type":"ContainerStarted","Data":"1c3cce391c8a2b4cbd0783f6ae9534ac8005b21774297ffbfa7fd7fbc544bca3"} Oct 30 07:00:03.224691 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:03.224600 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" podStartSLOduration=3.22458257 podStartE2EDuration="3.22458257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:03.223856556 +0000 UTC m=+4017.219813565" watchObservedRunningTime="2026-10-30 07:00:03.22458257 +0000 UTC m=+4017.220539582" Oct 30 07:00:03.266078 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:03.266021 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" podStartSLOduration=3.266007311 podStartE2EDuration="3.266007311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:03.262229588 +0000 UTC m=+4017.258186612" watchObservedRunningTime="2026-10-30 07:00:03.266007311 +0000 UTC m=+4017.261964320" Oct 30 07:00:03.306568 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:03.306517 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" podStartSLOduration=3.306500177 podStartE2EDuration="3.306500177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:03.301741807 +0000 UTC m=+4017.297698819" watchObservedRunningTime="2026-10-30 07:00:03.306500177 +0000 UTC m=+4017.302457247" Oct 30 07:00:03.341143 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:03.341087 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" podStartSLOduration=3.341068422 podStartE2EDuration="3.341068422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:03.339419082 +0000 UTC m=+4017.335376092" watchObservedRunningTime="2026-10-30 07:00:03.341068422 +0000 UTC m=+4017.337025431" Oct 30 07:00:03.528992 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:03.528953 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5"] Oct 30 07:00:03.532500 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:03.532463 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-30fbb6-ql9r5"] Oct 30 07:00:04.120741 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.120696 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ee0f28-85c1-41ad-9d4b-03e216bd641e" path="/var/lib/kubelet/pods/f0ee0f28-85c1-41ad-9d4b-03e216bd641e/volumes" Oct 30 07:00:04.207788 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.206654 5001 generic.go:356] "Generic (PLEG): container finished" podID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerID="1c3cce391c8a2b4cbd0783f6ae9534ac8005b21774297ffbfa7fd7fbc544bca3" exitCode=60 Oct 30 07:00:04.207788 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.207002 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" event={"ID":"e45c44da-7337-4c2e-8a79-f6e76e853db6","Type":"ContainerDied","Data":"1c3cce391c8a2b4cbd0783f6ae9534ac8005b21774297ffbfa7fd7fbc544bca3"} Oct 30 07:00:04.207788 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.207058 5001 scope.go:122] "RemoveContainer" containerID="ae9eed2e65e60fc40b39f8d159ef9d3166e0f02548f71d1e24d76b7126ce6a60" Oct 30 07:00:04.207788 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.207162 5001 scope.go:122] "RemoveContainer" containerID="1c3cce391c8a2b4cbd0783f6ae9534ac8005b21774297ffbfa7fd7fbc544bca3" Oct 30 07:00:04.207788 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:04.207460 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr_odh-ai-gateway-infra(e45c44da-7337-4c2e-8a79-f6e76e853db6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" Oct 30 07:00:04.210378 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.210158 5001 generic.go:356] "Generic (PLEG): container finished" podID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerID="bc6ddda41f611f653185b2753f66b013faeea2d5545137c848de14e14a6c1004" exitCode=60 Oct 30 07:00:04.210891 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.210585 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" event={"ID":"0df149d0-8b5f-4273-aad4-11a641230f8d","Type":"ContainerDied","Data":"bc6ddda41f611f653185b2753f66b013faeea2d5545137c848de14e14a6c1004"} Oct 30 07:00:04.210891 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.210621 5001 scope.go:122] "RemoveContainer" containerID="bc6ddda41f611f653185b2753f66b013faeea2d5545137c848de14e14a6c1004" Oct 30 07:00:04.210891 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:04.210857 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml_odh-ai-gateway-infra(0df149d0-8b5f-4273-aad4-11a641230f8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" Oct 30 07:00:04.212755 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.212584 5001 generic.go:356] "Generic (PLEG): container finished" podID="5d03a7c2-0012-42dd-afd3-03c0d3c468e3" containerID="cb0dc7699158a55cbeb4dc11003206fa8bbdc59eae46e3c362d7f7d5d52caee1" exitCode=0 Oct 30 07:00:04.212755 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.212697 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" event={"ID":"5d03a7c2-0012-42dd-afd3-03c0d3c468e3","Type":"ContainerDied","Data":"cb0dc7699158a55cbeb4dc11003206fa8bbdc59eae46e3c362d7f7d5d52caee1"} Oct 30 07:00:04.219106 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.218618 5001 generic.go:356] "Generic (PLEG): container finished" podID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerID="fb62a3893ae45e1346459b4ccd4c7a40160a4fe871775230b0cca226f4ff39e5" exitCode=60 Oct 30 07:00:04.219106 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.218705 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" event={"ID":"56d1c8a0-de78-4cc5-85e5-f538ce6237ba","Type":"ContainerDied","Data":"fb62a3893ae45e1346459b4ccd4c7a40160a4fe871775230b0cca226f4ff39e5"} Oct 30 07:00:04.219106 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.219005 5001 scope.go:122] "RemoveContainer" containerID="fb62a3893ae45e1346459b4ccd4c7a40160a4fe871775230b0cca226f4ff39e5" Oct 30 07:00:04.221614 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:04.219448 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh_odh-ai-gateway-infra(56d1c8a0-de78-4cc5-85e5-f538ce6237ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" Oct 30 07:00:04.223205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.223185 5001 generic.go:356] "Generic (PLEG): container finished" podID="1131c14c-64e3-4840-88b7-50b0bde68434" containerID="e4cf4464467e1b947aad230ab5e3cfa63fed9bfd34b6bc80eaf02b9c062ab25d" exitCode=0 Oct 30 07:00:04.223383 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.223229 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" event={"ID":"1131c14c-64e3-4840-88b7-50b0bde68434","Type":"ContainerDied","Data":"e4cf4464467e1b947aad230ab5e3cfa63fed9bfd34b6bc80eaf02b9c062ab25d"} Oct 30 07:00:04.229909 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.229801 5001 scope.go:122] "RemoveContainer" containerID="f09531459141aacd4a30beac63406e2211cbca7edb86b91f88d8e8337dac6b9d" Oct 30 07:00:04.272087 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.272051 5001 scope.go:122] "RemoveContainer" containerID="be81e5c687abafddda1148b8c1f15356a60ff11d7b4e93f4c1d02d3f0e22ef62" Oct 30 07:00:04.275167 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.275137 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 07:00:04.301392 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.301365 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 07:00:04.391298 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.391254 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-kube-api-access-5pvlq\" (UniqueName: \"kubernetes.io/projected/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-kube-api-access-5pvlq\") pod \"5d03a7c2-0012-42dd-afd3-03c0d3c468e3\" (UID: \"5d03a7c2-0012-42dd-afd3-03c0d3c468e3\") " Oct 30 07:00:04.391521 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.391383 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1131c14c-64e3-4840-88b7-50b0bde68434-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1131c14c-64e3-4840-88b7-50b0bde68434-plugins-config-volume\") pod \"1131c14c-64e3-4840-88b7-50b0bde68434\" (UID: \"1131c14c-64e3-4840-88b7-50b0bde68434\") " Oct 30 07:00:04.391521 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.391441 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-plugins-config-volume\") pod \"5d03a7c2-0012-42dd-afd3-03c0d3c468e3\" (UID: \"5d03a7c2-0012-42dd-afd3-03c0d3c468e3\") " Oct 30 07:00:04.391521 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.391509 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1131c14c-64e3-4840-88b7-50b0bde68434-kube-api-access-sdfpg\" (UniqueName: \"kubernetes.io/projected/1131c14c-64e3-4840-88b7-50b0bde68434-kube-api-access-sdfpg\") pod \"1131c14c-64e3-4840-88b7-50b0bde68434\" (UID: \"1131c14c-64e3-4840-88b7-50b0bde68434\") " Oct 30 07:00:04.392485 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.392408 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1131c14c-64e3-4840-88b7-50b0bde68434-plugins-config-volume" pod "1131c14c-64e3-4840-88b7-50b0bde68434" (UID: "1131c14c-64e3-4840-88b7-50b0bde68434"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:04.392845 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.392819 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-plugins-config-volume" pod "5d03a7c2-0012-42dd-afd3-03c0d3c468e3" (UID: "5d03a7c2-0012-42dd-afd3-03c0d3c468e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:04.394927 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.394902 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-kube-api-access-5pvlq" pod "5d03a7c2-0012-42dd-afd3-03c0d3c468e3" (UID: "5d03a7c2-0012-42dd-afd3-03c0d3c468e3"). InnerVolumeSpecName "kube-api-access-5pvlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:04.395038 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.394939 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1131c14c-64e3-4840-88b7-50b0bde68434-kube-api-access-sdfpg" pod "1131c14c-64e3-4840-88b7-50b0bde68434" (UID: "1131c14c-64e3-4840-88b7-50b0bde68434"). InnerVolumeSpecName "kube-api-access-sdfpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:04.492568 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.492484 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:04.492568 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.492517 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdfpg\" (UniqueName: \"kubernetes.io/projected/1131c14c-64e3-4840-88b7-50b0bde68434-kube-api-access-sdfpg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:04.492568 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.492528 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pvlq\" (UniqueName: \"kubernetes.io/projected/5d03a7c2-0012-42dd-afd3-03c0d3c468e3-kube-api-access-5pvlq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:04.492568 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.492536 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1131c14c-64e3-4840-88b7-50b0bde68434-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:04.977880 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:04.977846 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 07:00:05.042062 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.042020 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 07:00:05.114485 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.114443 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-plugins-config-volume\") pod \"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4\" (UID: \"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4\") " Oct 30 07:00:05.114715 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.114558 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-kube-api-access-52ksq\" (UniqueName: \"kubernetes.io/projected/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-kube-api-access-52ksq\") pod \"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4\" (UID: \"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4\") " Oct 30 07:00:05.114715 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.114690 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80eaafa-a522-400a-802f-0a54e846cc3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80eaafa-a522-400a-802f-0a54e846cc3d-plugins-config-volume\") pod \"e80eaafa-a522-400a-802f-0a54e846cc3d\" (UID: \"e80eaafa-a522-400a-802f-0a54e846cc3d\") " Oct 30 07:00:05.114863 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.114743 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80eaafa-a522-400a-802f-0a54e846cc3d-kube-api-access-rfn5q\" (UniqueName: \"kubernetes.io/projected/e80eaafa-a522-400a-802f-0a54e846cc3d-kube-api-access-rfn5q\") pod \"e80eaafa-a522-400a-802f-0a54e846cc3d\" (UID: \"e80eaafa-a522-400a-802f-0a54e846cc3d\") " Oct 30 07:00:05.114930 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.114910 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-plugins-config-volume" pod "6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4" (UID: "6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:05.115566 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.115539 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80eaafa-a522-400a-802f-0a54e846cc3d-plugins-config-volume" pod "e80eaafa-a522-400a-802f-0a54e846cc3d" (UID: "e80eaafa-a522-400a-802f-0a54e846cc3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:05.115727 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.115701 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:05.115794 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.115731 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80eaafa-a522-400a-802f-0a54e846cc3d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:05.117728 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.117703 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80eaafa-a522-400a-802f-0a54e846cc3d-kube-api-access-rfn5q" pod "e80eaafa-a522-400a-802f-0a54e846cc3d" (UID: "e80eaafa-a522-400a-802f-0a54e846cc3d"). InnerVolumeSpecName "kube-api-access-rfn5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:05.117876 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.117856 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-kube-api-access-52ksq" pod "6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4" (UID: "6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4"). InnerVolumeSpecName "kube-api-access-52ksq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:05.216910 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.216866 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfn5q\" (UniqueName: \"kubernetes.io/projected/e80eaafa-a522-400a-802f-0a54e846cc3d-kube-api-access-rfn5q\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:05.216910 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.216904 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52ksq\" (UniqueName: \"kubernetes.io/projected/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4-kube-api-access-52ksq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:05.232234 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.232204 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" Oct 30 07:00:05.232234 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.232227 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2" event={"ID":"1131c14c-64e3-4840-88b7-50b0bde68434","Type":"ContainerDied","Data":"bed771c4dd508d862ec5d98d29c311cba479f502d76b7bd22dd7eec1dc46d9b4"} Oct 30 07:00:05.232538 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.232295 5001 scope.go:122] "RemoveContainer" containerID="e4cf4464467e1b947aad230ab5e3cfa63fed9bfd34b6bc80eaf02b9c062ab25d" Oct 30 07:00:05.234568 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.234550 5001 scope.go:122] "RemoveContainer" containerID="1c3cce391c8a2b4cbd0783f6ae9534ac8005b21774297ffbfa7fd7fbc544bca3" Oct 30 07:00:05.234819 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:05.234793 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr_odh-ai-gateway-infra(e45c44da-7337-4c2e-8a79-f6e76e853db6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" Oct 30 07:00:05.236415 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.236392 5001 scope.go:122] "RemoveContainer" containerID="bc6ddda41f611f653185b2753f66b013faeea2d5545137c848de14e14a6c1004" Oct 30 07:00:05.236683 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:05.236659 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml_odh-ai-gateway-infra(0df149d0-8b5f-4273-aad4-11a641230f8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" Oct 30 07:00:05.237633 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.237615 5001 generic.go:356] "Generic (PLEG): container finished" podID="e80eaafa-a522-400a-802f-0a54e846cc3d" containerID="f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb" exitCode=0 Oct 30 07:00:05.237737 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.237659 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" Oct 30 07:00:05.237737 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.237721 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" event={"ID":"e80eaafa-a522-400a-802f-0a54e846cc3d","Type":"ContainerDied","Data":"f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb"} Oct 30 07:00:05.237854 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.237775 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx" event={"ID":"e80eaafa-a522-400a-802f-0a54e846cc3d","Type":"ContainerDied","Data":"a9516232ffc25515ab4d8f43f09ed64f32fe8daee2165c4ec303559d4b4cf0c9"} Oct 30 07:00:05.239316 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.239279 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" Oct 30 07:00:05.239424 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.239322 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5" event={"ID":"5d03a7c2-0012-42dd-afd3-03c0d3c468e3","Type":"ContainerDied","Data":"6fffe45d4e3029e8f2179cb6144962b5f8b1d0501e2ca4efbfad102756e07c50"} Oct 30 07:00:05.247343 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.242872 5001 generic.go:356] "Generic (PLEG): container finished" podID="6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4" containerID="21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e" exitCode=0 Oct 30 07:00:05.247343 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.242943 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" event={"ID":"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4","Type":"ContainerDied","Data":"21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e"} Oct 30 07:00:05.247343 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.242976 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" event={"ID":"6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4","Type":"ContainerDied","Data":"7c5613c3be1690d156569c072b286ef35d2d458e5a86694327ceb970b083915e"} Oct 30 07:00:05.247343 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.243039 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9" Oct 30 07:00:05.247850 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.247827 5001 scope.go:122] "RemoveContainer" containerID="fb62a3893ae45e1346459b4ccd4c7a40160a4fe871775230b0cca226f4ff39e5" Oct 30 07:00:05.248127 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:05.248099 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh_odh-ai-gateway-infra(56d1c8a0-de78-4cc5-85e5-f538ce6237ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" Oct 30 07:00:05.251065 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.251044 5001 scope.go:122] "RemoveContainer" containerID="f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb" Oct 30 07:00:05.265406 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.265386 5001 scope.go:122] "RemoveContainer" containerID="f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb" Oct 30 07:00:05.265738 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:05.265720 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb\": container with ID starting with f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb not found: ID does not exist" containerID="f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb" Oct 30 07:00:05.265788 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.265750 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb"} err="failed to get container status \"f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb\": rpc error: code = NotFound desc = could not find container \"f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb\": container with ID starting with f80df3c3a0f096990bcb0c46d03603df854006f005116074bf7143318bdcbceb not found: ID does not exist" Oct 30 07:00:05.265788 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.265767 5001 scope.go:122] "RemoveContainer" containerID="cb0dc7699158a55cbeb4dc11003206fa8bbdc59eae46e3c362d7f7d5d52caee1" Oct 30 07:00:05.277272 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.277250 5001 scope.go:122] "RemoveContainer" containerID="21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e" Oct 30 07:00:05.288135 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.288114 5001 scope.go:122] "RemoveContainer" containerID="21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e" Oct 30 07:00:05.288524 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:05.288499 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e\": container with ID starting with 21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e not found: ID does not exist" containerID="21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e" Oct 30 07:00:05.288632 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.288531 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e"} err="failed to get container status \"21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e\": rpc error: code = NotFound desc = could not find container \"21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e\": container with ID starting with 21b808f73ebe71e77d7ee9f1aa4c2766ad85280051d14ec08eaeb1f374eb233e not found: ID does not exist" Oct 30 07:00:05.450037 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.449977 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx"] Oct 30 07:00:05.453766 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.453717 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-adcdcb-754787d6fb-98djx"] Oct 30 07:00:05.485491 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.485461 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5"] Oct 30 07:00:05.492355 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.492320 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-645f2b-d56dddf5f-b97d5"] Oct 30 07:00:05.525848 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.525811 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2"] Oct 30 07:00:05.532334 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.532292 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-645f2b-cd8c4b676-jq6b2"] Oct 30 07:00:05.565002 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.564918 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9"] Oct 30 07:00:05.570696 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:05.570669 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-adcdcb-7b79b5cf47-zhhc9"] Oct 30 07:00:06.118492 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:06.118462 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1131c14c-64e3-4840-88b7-50b0bde68434" path="/var/lib/kubelet/pods/1131c14c-64e3-4840-88b7-50b0bde68434/volumes" Oct 30 07:00:06.118857 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:06.118844 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d03a7c2-0012-42dd-afd3-03c0d3c468e3" path="/var/lib/kubelet/pods/5d03a7c2-0012-42dd-afd3-03c0d3c468e3/volumes" Oct 30 07:00:06.119184 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:06.119172 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4" path="/var/lib/kubelet/pods/6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4/volumes" Oct 30 07:00:06.119532 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:06.119519 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80eaafa-a522-400a-802f-0a54e846cc3d" path="/var/lib/kubelet/pods/e80eaafa-a522-400a-802f-0a54e846cc3d/volumes" Oct 30 07:00:07.026059 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:07.026004 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" podUID="0b05b248-f62b-466f-b923-311b6d99c48e" containerName="payload-pre-processing" containerID="cri-o://550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0" gracePeriod=25 Oct 30 07:00:07.489229 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:07.489180 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6"] Oct 30 07:00:07.489503 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:07.489472 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" containerID="cri-o://e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c" gracePeriod=30 Oct 30 07:00:08.026759 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.026686 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 07:00:08.150961 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.150923 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-data\") pod \"4ad31106-b066-48b3-9bc4-a8e442cb719e\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " Oct 30 07:00:08.150961 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.150964 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-envoy\") pod \"4ad31106-b066-48b3-9bc4-a8e442cb719e\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " Oct 30 07:00:08.151213 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.151012 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-token\" (UniqueName: \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-token\") pod \"4ad31106-b066-48b3-9bc4-a8e442cb719e\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " Oct 30 07:00:08.151213 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.151083 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-podinfo\") pod \"4ad31106-b066-48b3-9bc4-a8e442cb719e\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " Oct 30 07:00:08.151213 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.151138 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-kube-api-access-nxw4b\" (UniqueName: \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-kube-api-access-nxw4b\") pod \"4ad31106-b066-48b3-9bc4-a8e442cb719e\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " Oct 30 07:00:08.151213 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.151175 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-certs\") pod \"4ad31106-b066-48b3-9bc4-a8e442cb719e\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " Oct 30 07:00:08.151495 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.151215 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-credential-socket\") pod \"4ad31106-b066-48b3-9bc4-a8e442cb719e\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " Oct 30 07:00:08.151495 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.151270 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-data" pod "4ad31106-b066-48b3-9bc4-a8e442cb719e" (UID: "4ad31106-b066-48b3-9bc4-a8e442cb719e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:08.151495 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.151291 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-socket\") pod \"4ad31106-b066-48b3-9bc4-a8e442cb719e\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " Oct 30 07:00:08.151731 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.151514 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-certs" pod "4ad31106-b066-48b3-9bc4-a8e442cb719e" (UID: "4ad31106-b066-48b3-9bc4-a8e442cb719e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:08.151731 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.151563 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-credential-socket" pod "4ad31106-b066-48b3-9bc4-a8e442cb719e" (UID: "4ad31106-b066-48b3-9bc4-a8e442cb719e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:08.151731 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.151626 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-socket" pod "4ad31106-b066-48b3-9bc4-a8e442cb719e" (UID: "4ad31106-b066-48b3-9bc4-a8e442cb719e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:08.151731 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.151721 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad31106-b066-48b3-9bc4-a8e442cb719e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad31106-b066-48b3-9bc4-a8e442cb719e-istiod-ca-cert\") pod \"4ad31106-b066-48b3-9bc4-a8e442cb719e\" (UID: \"4ad31106-b066-48b3-9bc4-a8e442cb719e\") " Oct 30 07:00:08.152212 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.152188 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad31106-b066-48b3-9bc4-a8e442cb719e-istiod-ca-cert" pod "4ad31106-b066-48b3-9bc4-a8e442cb719e" (UID: "4ad31106-b066-48b3-9bc4-a8e442cb719e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:08.152422 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.152401 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:08.152510 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.152429 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:08.152510 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.152447 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:08.152510 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.152461 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad31106-b066-48b3-9bc4-a8e442cb719e-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:08.152510 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.152475 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:08.153766 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.153740 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-token" pod "4ad31106-b066-48b3-9bc4-a8e442cb719e" (UID: "4ad31106-b066-48b3-9bc4-a8e442cb719e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:08.153766 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.153752 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-podinfo" pod "4ad31106-b066-48b3-9bc4-a8e442cb719e" (UID: "4ad31106-b066-48b3-9bc4-a8e442cb719e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:00:08.153922 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.153825 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-kube-api-access-nxw4b" pod "4ad31106-b066-48b3-9bc4-a8e442cb719e" (UID: "4ad31106-b066-48b3-9bc4-a8e442cb719e"). InnerVolumeSpecName "kube-api-access-nxw4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:08.154157 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.154137 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-envoy" pod "4ad31106-b066-48b3-9bc4-a8e442cb719e" (UID: "4ad31106-b066-48b3-9bc4-a8e442cb719e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:08.253442 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.253407 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:08.253442 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.253441 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:08.253442 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.253453 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad31106-b066-48b3-9bc4-a8e442cb719e-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:08.253760 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.253466 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxw4b\" (UniqueName: \"kubernetes.io/projected/4ad31106-b066-48b3-9bc4-a8e442cb719e-kube-api-access-nxw4b\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:08.266378 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:08.266342 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 07:00:08.279169 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.279068 5001 generic.go:356] "Generic (PLEG): container finished" podID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerID="a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada" exitCode=0 Oct 30 07:00:08.279398 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.279198 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" Oct 30 07:00:08.279398 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.279256 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" event={"ID":"4ad31106-b066-48b3-9bc4-a8e442cb719e","Type":"ContainerDied","Data":"a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada"} Oct 30 07:00:08.279398 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.279324 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26" event={"ID":"4ad31106-b066-48b3-9bc4-a8e442cb719e","Type":"ContainerDied","Data":"ec64112b1c69fa87d32cf01336736f2089f696598bbb7b0bde396f8aae185bdd"} Oct 30 07:00:08.279398 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.279358 5001 scope.go:122] "RemoveContainer" containerID="a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada" Oct 30 07:00:08.292199 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.292173 5001 scope.go:122] "RemoveContainer" containerID="7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c" Oct 30 07:00:08.303292 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.303271 5001 scope.go:122] "RemoveContainer" containerID="a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada" Oct 30 07:00:08.303620 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:08.303600 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada\": container with ID starting with a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada not found: ID does not exist" containerID="a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada" Oct 30 07:00:08.303687 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.303630 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada"} err="failed to get container status \"a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada\": rpc error: code = NotFound desc = could not find container \"a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada\": container with ID starting with a5b444b9bdbd95aa8c547cebee82ee935cfc6c78b20719182b7dd22cbf884ada not found: ID does not exist" Oct 30 07:00:08.303687 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.303646 5001 scope.go:122] "RemoveContainer" containerID="7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c" Oct 30 07:00:08.303967 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:08.303948 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c\": container with ID starting with 7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c not found: ID does not exist" containerID="7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c" Oct 30 07:00:08.304025 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.303971 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c"} err="failed to get container status \"7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c\": rpc error: code = NotFound desc = could not find container \"7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c\": container with ID starting with 7fe48983564663316c26a0d329122bb54ed05975e3d24cfeb3e8ed077a12de2c not found: ID does not exist" Oct 30 07:00:08.325775 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.325733 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" podUID="0b05b248-f62b-466f-b923-311b6d99c48e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:08.339610 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.339578 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26"] Oct 30 07:00:08.342757 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:08.342730 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-adcdcb-openshift-default-85679766c5-7zq26"] Oct 30 07:00:10.117404 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:10.117364 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" path="/var/lib/kubelet/pods/4ad31106-b066-48b3-9bc4-a8e442cb719e/volumes" Oct 30 07:00:10.867283 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:10.867245 5001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:00:10.867491 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:10.867317 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:00:10.932819 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:10.932787 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx"] Oct 30 07:00:10.933088 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:10.933056 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" containerID="cri-o://83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d" gracePeriod=30 Oct 30 07:00:12.649356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.649318 5001 scope.go:122] "RemoveContainer" containerID="41f21610c93e74b22b9f8b6e5e4ab63d9608390310aacbb29b0ca38e81f976d3" Oct 30 07:00:12.661997 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.661973 5001 scope.go:122] "RemoveContainer" containerID="11905891ff322cb2d017bafb425aacf87cd8a3ec682b0de930bf80df10e752ec" Oct 30 07:00:12.679487 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.679213 5001 scope.go:122] "RemoveContainer" containerID="503312f059b8a987298c3f1c5dfa7cac086a596b2e34c2589709b6dbfb61512c" Oct 30 07:00:12.707496 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.707474 5001 scope.go:122] "RemoveContainer" containerID="8f8d8a72b497c19107c781c6352d12241667d37cb9ad670208d67ac5111c8ba8" Oct 30 07:00:12.720242 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.720217 5001 scope.go:122] "RemoveContainer" containerID="ef3712b296b4eb50ad980c4058413e22a5b9118a862f949fd77351c47bc94199" Oct 30 07:00:12.731631 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.731604 5001 scope.go:122] "RemoveContainer" containerID="0a0d70da3f71307de4f911204d39d6cd04d44bd2bff9d6ce2c961f25afb34078" Oct 30 07:00:12.742543 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.742516 5001 scope.go:122] "RemoveContainer" containerID="eeecf4d942dd2b7787bc3e1fe21c83a7e3a3f20161a54e67d4be68a2e6616657" Oct 30 07:00:12.793898 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.793871 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 07:00:12.909656 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.909629 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-token\" (UniqueName: \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-token\") pod \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " Oct 30 07:00:12.909742 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.909666 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-data\") pod \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " Oct 30 07:00:12.909742 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.909691 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-podinfo\") pod \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " Oct 30 07:00:12.909861 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.909768 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-certs\") pod \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " Oct 30 07:00:12.909861 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.909822 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-kube-api-access-668nv\" (UniqueName: \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-kube-api-access-668nv\") pod \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " Oct 30 07:00:12.909979 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.909871 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-credential-socket\") pod \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " Oct 30 07:00:12.910035 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.909992 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-data" pod "c741b57d-5029-4bfe-ab30-f58e5e45eb2c" (UID: "c741b57d-5029-4bfe-ab30-f58e5e45eb2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:12.910644 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.910112 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-certs" pod "c741b57d-5029-4bfe-ab30-f58e5e45eb2c" (UID: "c741b57d-5029-4bfe-ab30-f58e5e45eb2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:12.910644 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.910161 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-socket\") pod \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " Oct 30 07:00:12.910644 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.910201 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-envoy\") pod \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " Oct 30 07:00:12.910644 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.910240 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istiod-ca-cert\") pod \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\" (UID: \"c741b57d-5029-4bfe-ab30-f58e5e45eb2c\") " Oct 30 07:00:12.910644 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.910381 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-credential-socket" pod "c741b57d-5029-4bfe-ab30-f58e5e45eb2c" (UID: "c741b57d-5029-4bfe-ab30-f58e5e45eb2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:12.910644 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.910596 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-socket" pod "c741b57d-5029-4bfe-ab30-f58e5e45eb2c" (UID: "c741b57d-5029-4bfe-ab30-f58e5e45eb2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:12.911173 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.910812 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:12.911173 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.910832 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:12.911173 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.910857 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:12.911173 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.910870 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:12.911173 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.911058 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istiod-ca-cert" pod "c741b57d-5029-4bfe-ab30-f58e5e45eb2c" (UID: "c741b57d-5029-4bfe-ab30-f58e5e45eb2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:12.913019 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.912614 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-kube-api-access-668nv" pod "c741b57d-5029-4bfe-ab30-f58e5e45eb2c" (UID: "c741b57d-5029-4bfe-ab30-f58e5e45eb2c"). InnerVolumeSpecName "kube-api-access-668nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:12.913019 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.912902 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-podinfo" pod "c741b57d-5029-4bfe-ab30-f58e5e45eb2c" (UID: "c741b57d-5029-4bfe-ab30-f58e5e45eb2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:00:12.913431 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.913410 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-envoy" pod "c741b57d-5029-4bfe-ab30-f58e5e45eb2c" (UID: "c741b57d-5029-4bfe-ab30-f58e5e45eb2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:12.913516 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:12.913461 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-token" pod "c741b57d-5029-4bfe-ab30-f58e5e45eb2c" (UID: "c741b57d-5029-4bfe-ab30-f58e5e45eb2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:13.011553 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.011515 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:13.011553 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.011543 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:13.011553 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.011552 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-668nv\" (UniqueName: \"kubernetes.io/projected/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-kube-api-access-668nv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:13.011553 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.011562 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:13.011977 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.011573 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c741b57d-5029-4bfe-ab30-f58e5e45eb2c-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:13.014426 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.014397 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx"] Oct 30 07:00:13.325478 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.325393 5001 generic.go:356] "Generic (PLEG): container finished" podID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerID="e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c" exitCode=0 Oct 30 07:00:13.325478 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.325445 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" Oct 30 07:00:13.325731 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.325479 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" event={"ID":"c741b57d-5029-4bfe-ab30-f58e5e45eb2c","Type":"ContainerDied","Data":"e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c"} Oct 30 07:00:13.325731 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.325542 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6" event={"ID":"c741b57d-5029-4bfe-ab30-f58e5e45eb2c","Type":"ContainerDied","Data":"c637dbbb344dcc6186cb00dfa61cf4f4078bb8ebf8993b48d5114eed57c41698"} Oct 30 07:00:13.325731 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.325579 5001 scope.go:122] "RemoveContainer" containerID="e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c" Oct 30 07:00:13.337642 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.337623 5001 scope.go:122] "RemoveContainer" containerID="e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c" Oct 30 07:00:13.337935 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:13.337917 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c\": container with ID starting with e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c not found: ID does not exist" containerID="e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c" Oct 30 07:00:13.337983 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.337944 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c"} err="failed to get container status \"e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c\": rpc error: code = NotFound desc = could not find container \"e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c\": container with ID starting with e84685754a5173398959d94931c6aaeb651ac9a7ea93a062662a4eb566950f4c not found: ID does not exist" Oct 30 07:00:13.376581 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.376547 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6"] Oct 30 07:00:13.384927 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:13.384896 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-868b1f-openshift-default-85db5d8bc9-tjtn6"] Oct 30 07:00:14.118824 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:14.118790 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" path="/var/lib/kubelet/pods/c741b57d-5029-4bfe-ab30-f58e5e45eb2c/volumes" Oct 30 07:00:14.144175 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:14.144140 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn"] Oct 30 07:00:16.115066 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.115038 5001 scope.go:122] "RemoveContainer" containerID="bc6ddda41f611f653185b2753f66b013faeea2d5545137c848de14e14a6c1004" Oct 30 07:00:16.115524 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.115172 5001 scope.go:122] "RemoveContainer" containerID="1c3cce391c8a2b4cbd0783f6ae9534ac8005b21774297ffbfa7fd7fbc544bca3" Oct 30 07:00:16.220763 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.220740 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 07:00:16.350074 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350044 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-istio-token\" (UniqueName: \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-istio-token\") pod \"4549b201-684a-49f1-b871-4c0866e27bbb\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " Oct 30 07:00:16.350286 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350106 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-credential-socket\") pod \"4549b201-684a-49f1-b871-4c0866e27bbb\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " Oct 30 07:00:16.350286 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350161 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4549b201-684a-49f1-b871-4c0866e27bbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4549b201-684a-49f1-b871-4c0866e27bbb-istio-podinfo\") pod \"4549b201-684a-49f1-b871-4c0866e27bbb\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " Oct 30 07:00:16.350286 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350208 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-certs\") pod \"4549b201-684a-49f1-b871-4c0866e27bbb\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " Oct 30 07:00:16.350286 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350232 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4549b201-684a-49f1-b871-4c0866e27bbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4549b201-684a-49f1-b871-4c0866e27bbb-istiod-ca-cert\") pod \"4549b201-684a-49f1-b871-4c0866e27bbb\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " Oct 30 07:00:16.350286 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350251 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-socket\") pod \"4549b201-684a-49f1-b871-4c0866e27bbb\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " Oct 30 07:00:16.350624 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350292 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-envoy\") pod \"4549b201-684a-49f1-b871-4c0866e27bbb\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " Oct 30 07:00:16.350624 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350341 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-data\") pod \"4549b201-684a-49f1-b871-4c0866e27bbb\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " Oct 30 07:00:16.350624 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350449 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-kube-api-access-gjgv2\" (UniqueName: \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-kube-api-access-gjgv2\") pod \"4549b201-684a-49f1-b871-4c0866e27bbb\" (UID: \"4549b201-684a-49f1-b871-4c0866e27bbb\") " Oct 30 07:00:16.350624 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350515 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-credential-socket" pod "4549b201-684a-49f1-b871-4c0866e27bbb" (UID: "4549b201-684a-49f1-b871-4c0866e27bbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:16.350850 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350724 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-socket" pod "4549b201-684a-49f1-b871-4c0866e27bbb" (UID: "4549b201-684a-49f1-b871-4c0866e27bbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:16.350850 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350737 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-certs" pod "4549b201-684a-49f1-b871-4c0866e27bbb" (UID: "4549b201-684a-49f1-b871-4c0866e27bbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:16.350850 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350669 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4549b201-684a-49f1-b871-4c0866e27bbb-istiod-ca-cert" pod "4549b201-684a-49f1-b871-4c0866e27bbb" (UID: "4549b201-684a-49f1-b871-4c0866e27bbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:16.350850 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350800 5001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:16.350850 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.350842 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-data" pod "4549b201-684a-49f1-b871-4c0866e27bbb" (UID: "4549b201-684a-49f1-b871-4c0866e27bbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:16.353373 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.352782 5001 generic.go:356] "Generic (PLEG): container finished" podID="4549b201-684a-49f1-b871-4c0866e27bbb" containerID="83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d" exitCode=0 Oct 30 07:00:16.353373 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.353037 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" event={"ID":"4549b201-684a-49f1-b871-4c0866e27bbb","Type":"ContainerDied","Data":"83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d"} Oct 30 07:00:16.353373 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.353092 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" event={"ID":"4549b201-684a-49f1-b871-4c0866e27bbb","Type":"ContainerDied","Data":"fe73c1492a88b04d23df40ee6ac981e490824f793330941766264bc2afac2741"} Oct 30 07:00:16.353373 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.353120 5001 scope.go:122] "RemoveContainer" containerID="83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d" Oct 30 07:00:16.353373 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.353268 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx" Oct 30 07:00:16.353693 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.353478 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-kube-api-access-gjgv2" pod "4549b201-684a-49f1-b871-4c0866e27bbb" (UID: "4549b201-684a-49f1-b871-4c0866e27bbb"). InnerVolumeSpecName "kube-api-access-gjgv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:16.353693 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.353619 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-envoy" pod "4549b201-684a-49f1-b871-4c0866e27bbb" (UID: "4549b201-684a-49f1-b871-4c0866e27bbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:16.353837 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.353815 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4549b201-684a-49f1-b871-4c0866e27bbb-istio-podinfo" pod "4549b201-684a-49f1-b871-4c0866e27bbb" (UID: "4549b201-684a-49f1-b871-4c0866e27bbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:00:16.353953 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.353937 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-istio-token" pod "4549b201-684a-49f1-b871-4c0866e27bbb" (UID: "4549b201-684a-49f1-b871-4c0866e27bbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:16.380458 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.380432 5001 scope.go:122] "RemoveContainer" containerID="f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7" Oct 30 07:00:16.396932 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.396855 5001 scope.go:122] "RemoveContainer" containerID="83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d" Oct 30 07:00:16.397260 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:16.397239 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d\": container with ID starting with 83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d not found: ID does not exist" containerID="83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d" Oct 30 07:00:16.397384 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.397272 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d"} err="failed to get container status \"83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d\": rpc error: code = NotFound desc = could not find container \"83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d\": container with ID starting with 83032a25fa9d12300070f2f46a541dce0557860dcbad794a6aa21ae68313137d not found: ID does not exist" Oct 30 07:00:16.397384 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.397289 5001 scope.go:122] "RemoveContainer" containerID="f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7" Oct 30 07:00:16.397597 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:16.397574 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7\": container with ID starting with f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7 not found: ID does not exist" containerID="f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7" Oct 30 07:00:16.397649 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.397604 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7"} err="failed to get container status \"f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7\": rpc error: code = NotFound desc = could not find container \"f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7\": container with ID starting with f02bbe2782d1c3ec05cfacfbf2dbe0184800825caded77c24655e1e9d8640ea7 not found: ID does not exist" Oct 30 07:00:16.422893 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.422861 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx"] Oct 30 07:00:16.429520 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.429473 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g"] Oct 30 07:00:16.432441 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.432413 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-30fbb6-openshift-default-df9656985-czkfx"] Oct 30 07:00:16.452274 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.452239 5001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:16.452274 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.452267 5001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4549b201-684a-49f1-b871-4c0866e27bbb-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:16.452274 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.452275 5001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:16.452626 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.452284 5001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:16.452626 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.452295 5001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4549b201-684a-49f1-b871-4c0866e27bbb-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:16.452626 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.452321 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjgv2\" (UniqueName: \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-kube-api-access-gjgv2\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:16.452626 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.452334 5001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4549b201-684a-49f1-b871-4c0866e27bbb-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:16.452626 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:16.452346 5001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4549b201-684a-49f1-b871-4c0866e27bbb-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:17.109653 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:17.109624 5001 scope.go:122] "RemoveContainer" containerID="fb62a3893ae45e1346459b4ccd4c7a40160a4fe871775230b0cca226f4ff39e5" Oct 30 07:00:17.362675 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:17.362645 5001 generic.go:356] "Generic (PLEG): container finished" podID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerID="59d9be9e74ac8dd743cac1f0784ad261205bbbd27a0b89cb1e3a4569043dbab6" exitCode=60 Oct 30 07:00:17.363122 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:17.362719 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" event={"ID":"e45c44da-7337-4c2e-8a79-f6e76e853db6","Type":"ContainerDied","Data":"59d9be9e74ac8dd743cac1f0784ad261205bbbd27a0b89cb1e3a4569043dbab6"} Oct 30 07:00:17.363122 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:17.362773 5001 scope.go:122] "RemoveContainer" containerID="1c3cce391c8a2b4cbd0783f6ae9534ac8005b21774297ffbfa7fd7fbc544bca3" Oct 30 07:00:17.363253 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:17.363146 5001 scope.go:122] "RemoveContainer" containerID="59d9be9e74ac8dd743cac1f0784ad261205bbbd27a0b89cb1e3a4569043dbab6" Oct 30 07:00:17.363464 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:17.363434 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr_odh-ai-gateway-infra(e45c44da-7337-4c2e-8a79-f6e76e853db6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" Oct 30 07:00:17.365074 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:17.365056 5001 generic.go:356] "Generic (PLEG): container finished" podID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerID="9bff41cbc7167163a39348e58a80a26317088aeee1b8afa6887a54b5451e0a7c" exitCode=60 Oct 30 07:00:17.365170 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:17.365128 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" event={"ID":"0df149d0-8b5f-4273-aad4-11a641230f8d","Type":"ContainerDied","Data":"9bff41cbc7167163a39348e58a80a26317088aeee1b8afa6887a54b5451e0a7c"} Oct 30 07:00:17.365650 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:17.365512 5001 scope.go:122] "RemoveContainer" containerID="9bff41cbc7167163a39348e58a80a26317088aeee1b8afa6887a54b5451e0a7c" Oct 30 07:00:17.365752 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:17.365736 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml_odh-ai-gateway-infra(0df149d0-8b5f-4273-aad4-11a641230f8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" Oct 30 07:00:17.376933 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:17.376915 5001 scope.go:122] "RemoveContainer" containerID="bc6ddda41f611f653185b2753f66b013faeea2d5545137c848de14e14a6c1004" Oct 30 07:00:18.118807 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.118766 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" path="/var/lib/kubelet/pods/4549b201-684a-49f1-b871-4c0866e27bbb/volumes" Oct 30 07:00:18.164871 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.164835 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr"] Oct 30 07:00:18.324336 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.324230 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" podUID="0b05b248-f62b-466f-b923-311b6d99c48e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:18.379280 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.379246 5001 generic.go:356] "Generic (PLEG): container finished" podID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerID="22a2a49938ff9bb7ca6fc113559facb78f85f1009972d2196c0eee2919bea240" exitCode=60 Oct 30 07:00:18.379803 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.379335 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" event={"ID":"56d1c8a0-de78-4cc5-85e5-f538ce6237ba","Type":"ContainerDied","Data":"22a2a49938ff9bb7ca6fc113559facb78f85f1009972d2196c0eee2919bea240"} Oct 30 07:00:18.379803 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.379405 5001 scope.go:122] "RemoveContainer" containerID="fb62a3893ae45e1346459b4ccd4c7a40160a4fe871775230b0cca226f4ff39e5" Oct 30 07:00:18.379937 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.379896 5001 scope.go:122] "RemoveContainer" containerID="22a2a49938ff9bb7ca6fc113559facb78f85f1009972d2196c0eee2919bea240" Oct 30 07:00:18.380168 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:18.380139 5001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh_odh-ai-gateway-infra(56d1c8a0-de78-4cc5-85e5-f538ce6237ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" Oct 30 07:00:18.454731 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.454698 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml"] Oct 30 07:00:18.542204 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.542171 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" Oct 30 07:00:18.661299 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.661269 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" Oct 30 07:00:18.678753 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.678723 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45c44da-7337-4c2e-8a79-f6e76e853db6-kube-api-access-dzkq8\" (UniqueName: \"kubernetes.io/projected/e45c44da-7337-4c2e-8a79-f6e76e853db6-kube-api-access-dzkq8\") pod \"e45c44da-7337-4c2e-8a79-f6e76e853db6\" (UID: \"e45c44da-7337-4c2e-8a79-f6e76e853db6\") " Oct 30 07:00:18.678942 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.678865 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45c44da-7337-4c2e-8a79-f6e76e853db6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45c44da-7337-4c2e-8a79-f6e76e853db6-maas-api-service-ca\") pod \"e45c44da-7337-4c2e-8a79-f6e76e853db6\" (UID: \"e45c44da-7337-4c2e-8a79-f6e76e853db6\") " Oct 30 07:00:18.679286 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.679261 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45c44da-7337-4c2e-8a79-f6e76e853db6-maas-api-service-ca" pod "e45c44da-7337-4c2e-8a79-f6e76e853db6" (UID: "e45c44da-7337-4c2e-8a79-f6e76e853db6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:18.681290 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.681257 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45c44da-7337-4c2e-8a79-f6e76e853db6-kube-api-access-dzkq8" pod "e45c44da-7337-4c2e-8a79-f6e76e853db6" (UID: "e45c44da-7337-4c2e-8a79-f6e76e853db6"). InnerVolumeSpecName "kube-api-access-dzkq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:18.779588 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.779556 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df149d0-8b5f-4273-aad4-11a641230f8d-kube-api-access-zrj46\" (UniqueName: \"kubernetes.io/projected/0df149d0-8b5f-4273-aad4-11a641230f8d-kube-api-access-zrj46\") pod \"0df149d0-8b5f-4273-aad4-11a641230f8d\" (UID: \"0df149d0-8b5f-4273-aad4-11a641230f8d\") " Oct 30 07:00:18.779783 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.779743 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df149d0-8b5f-4273-aad4-11a641230f8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df149d0-8b5f-4273-aad4-11a641230f8d-maas-api-service-ca\") pod \"0df149d0-8b5f-4273-aad4-11a641230f8d\" (UID: \"0df149d0-8b5f-4273-aad4-11a641230f8d\") " Oct 30 07:00:18.780055 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.780038 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzkq8\" (UniqueName: \"kubernetes.io/projected/e45c44da-7337-4c2e-8a79-f6e76e853db6-kube-api-access-dzkq8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:18.780134 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.780059 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45c44da-7337-4c2e-8a79-f6e76e853db6-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:18.780134 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.780096 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df149d0-8b5f-4273-aad4-11a641230f8d-maas-api-service-ca" pod "0df149d0-8b5f-4273-aad4-11a641230f8d" (UID: "0df149d0-8b5f-4273-aad4-11a641230f8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:18.781926 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.781902 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df149d0-8b5f-4273-aad4-11a641230f8d-kube-api-access-zrj46" pod "0df149d0-8b5f-4273-aad4-11a641230f8d" (UID: "0df149d0-8b5f-4273-aad4-11a641230f8d"). InnerVolumeSpecName "kube-api-access-zrj46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:18.846745 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.846705 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh"] Oct 30 07:00:18.881480 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.881390 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df149d0-8b5f-4273-aad4-11a641230f8d-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:18.881480 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:18.881419 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrj46\" (UniqueName: \"kubernetes.io/projected/0df149d0-8b5f-4273-aad4-11a641230f8d-kube-api-access-zrj46\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:19.145201 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.145082 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" podUID="c6d8db02-a932-402c-aa37-4e51ac01c609" containerName="payload-processing" containerID="cri-o://168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28" gracePeriod=25 Oct 30 07:00:19.392880 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.392844 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" Oct 30 07:00:19.392880 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.392868 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr" event={"ID":"e45c44da-7337-4c2e-8a79-f6e76e853db6","Type":"ContainerDied","Data":"0743a90d79886f3f397e7d479a9b7447eec427bc5d021cd9d47d4b27a24367de"} Oct 30 07:00:19.393549 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.392941 5001 scope.go:122] "RemoveContainer" containerID="59d9be9e74ac8dd743cac1f0784ad261205bbbd27a0b89cb1e3a4569043dbab6" Oct 30 07:00:19.397381 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.397258 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" Oct 30 07:00:19.397381 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.397287 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml" event={"ID":"0df149d0-8b5f-4273-aad4-11a641230f8d","Type":"ContainerDied","Data":"65bdbaa0c77845f3cb2d42a15b69ba95b015bbdac92f4a7f2571f2016ca75be4"} Oct 30 07:00:19.416227 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.416196 5001 scope.go:122] "RemoveContainer" containerID="9bff41cbc7167163a39348e58a80a26317088aeee1b8afa6887a54b5451e0a7c" Oct 30 07:00:19.453915 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.453874 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh"] Oct 30 07:00:19.481325 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.481268 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr"] Oct 30 07:00:19.492885 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.492836 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889060-m5xwr"] Oct 30 07:00:19.523746 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.523716 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml"] Oct 30 07:00:19.528336 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.528283 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889060-jfpml"] Oct 30 07:00:19.618702 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.618676 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" Oct 30 07:00:19.790718 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.790622 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-kube-api-access-v48mb\" (UniqueName: \"kubernetes.io/projected/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-kube-api-access-v48mb\") pod \"56d1c8a0-de78-4cc5-85e5-f538ce6237ba\" (UID: \"56d1c8a0-de78-4cc5-85e5-f538ce6237ba\") " Oct 30 07:00:19.790718 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.790702 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-maas-api-service-ca\") pod \"56d1c8a0-de78-4cc5-85e5-f538ce6237ba\" (UID: \"56d1c8a0-de78-4cc5-85e5-f538ce6237ba\") " Oct 30 07:00:19.791098 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.791074 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-maas-api-service-ca" pod "56d1c8a0-de78-4cc5-85e5-f538ce6237ba" (UID: "56d1c8a0-de78-4cc5-85e5-f538ce6237ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:19.791198 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.791184 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:19.793559 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.793530 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-kube-api-access-v48mb" pod "56d1c8a0-de78-4cc5-85e5-f538ce6237ba" (UID: "56d1c8a0-de78-4cc5-85e5-f538ce6237ba"). InnerVolumeSpecName "kube-api-access-v48mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:19.850845 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.850810 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" podUID="c6d8db02-a932-402c-aa37-4e51ac01c609" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:19.892262 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:19.892214 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v48mb\" (UniqueName: \"kubernetes.io/projected/56d1c8a0-de78-4cc5-85e5-f538ce6237ba-kube-api-access-v48mb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:20.117351 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:20.117289 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" path="/var/lib/kubelet/pods/0df149d0-8b5f-4273-aad4-11a641230f8d/volumes" Oct 30 07:00:20.117908 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:20.117890 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" path="/var/lib/kubelet/pods/e45c44da-7337-4c2e-8a79-f6e76e853db6/volumes" Oct 30 07:00:20.432111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:20.432014 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" event={"ID":"56d1c8a0-de78-4cc5-85e5-f538ce6237ba","Type":"ContainerDied","Data":"f36ab47524129d7e5d595dd03843f1888605bc6f66a6411109a4ce9ee3062ccd"} Oct 30 07:00:20.432111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:20.432080 5001 scope.go:122] "RemoveContainer" containerID="22a2a49938ff9bb7ca6fc113559facb78f85f1009972d2196c0eee2919bea240" Oct 30 07:00:20.432722 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:20.432026 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh" Oct 30 07:00:20.482168 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:20.482134 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh"] Oct 30 07:00:20.486280 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:20.486253 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8d6349-29889060-4trmh"] Oct 30 07:00:22.116850 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:22.116817 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" path="/var/lib/kubelet/pods/56d1c8a0-de78-4cc5-85e5-f538ce6237ba/volumes" Oct 30 07:00:22.251068 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:22.251037 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g"] Oct 30 07:00:23.133169 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:23.133133 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 07:00:23.847953 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:23.847897 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" podUID="0f4b00a4-8f73-4fa9-9e96-7100bce648a8" containerName="payload-pre-processing" containerID="cri-o://1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2" gracePeriod=25 Oct 30 07:00:24.264170 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264128 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p"] Oct 30 07:00:24.264883 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264863 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerName="cleanup" Oct 30 07:00:24.264946 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264887 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerName="cleanup" Oct 30 07:00:24.264946 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264900 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" Oct 30 07:00:24.264946 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264908 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" Oct 30 07:00:24.264946 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264920 5001 cpu_manager.go:414] "containerMap: removing container" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerName="cleanup" Oct 30 07:00:24.264946 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264928 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerName="cleanup" Oct 30 07:00:24.264946 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264946 5001 cpu_manager.go:414] "containerMap: removing container" podUID="1131c14c-64e3-4840-88b7-50b0bde68434" containerName="payload-processing" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264953 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1131c14c-64e3-4840-88b7-50b0bde68434" containerName="payload-processing" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264969 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e80eaafa-a522-400a-802f-0a54e846cc3d" containerName="payload-processing" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264976 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80eaafa-a522-400a-802f-0a54e846cc3d" containerName="payload-processing" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264986 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.264993 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265008 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265016 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265032 5001 cpu_manager.go:414] "containerMap: removing container" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265040 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265051 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265057 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265067 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265075 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265090 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265096 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265107 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265114 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265123 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265129 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265144 5001 cpu_manager.go:414] "containerMap: removing container" podUID="6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4" containerName="payload-pre-processing" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265151 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4" containerName="payload-pre-processing" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265166 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265173 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265201 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265208 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerName="cleanup" Oct 30 07:00:24.265205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265219 5001 cpu_manager.go:414] "containerMap: removing container" podUID="5d03a7c2-0012-42dd-afd3-03c0d3c468e3" containerName="payload-pre-processing" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265227 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d03a7c2-0012-42dd-afd3-03c0d3c468e3" containerName="payload-pre-processing" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265358 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265372 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerName="cleanup" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265381 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerName="cleanup" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265389 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab0ea91-b217-4fc4-9ed9-0c42fc48bfe4" containerName="payload-pre-processing" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265398 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265407 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265419 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerName="cleanup" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265427 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d03a7c2-0012-42dd-afd3-03c0d3c468e3" containerName="payload-pre-processing" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265436 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c741b57d-5029-4bfe-ab30-f58e5e45eb2c" containerName="istio-proxy" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265444 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerName="cleanup" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265452 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1131c14c-64e3-4840-88b7-50b0bde68434" containerName="payload-processing" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265463 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80eaafa-a522-400a-802f-0a54e846cc3d" containerName="payload-processing" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265472 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerName="cleanup" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265480 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerName="cleanup" Oct 30 07:00:24.266391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.265489 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerName="cleanup" Oct 30 07:00:24.272197 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.272174 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:24.288193 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.288166 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p"] Oct 30 07:00:24.365108 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.365078 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:00:24.441939 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.441907 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8szqb\" (UniqueName: \"kubernetes.io/projected/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-kube-api-access-8szqb\") pod \"maas-api-revoke-sub-d48a5223-q222p\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:24.442097 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.441948 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48a5223-q222p\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:24.442097 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.441982 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48a5223-q222p\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:24.543427 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.543328 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8szqb\" (UniqueName: \"kubernetes.io/projected/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-kube-api-access-8szqb\") pod \"maas-api-revoke-sub-d48a5223-q222p\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:24.543427 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.543391 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48a5223-q222p\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:24.543665 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.543440 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48a5223-q222p\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:24.544034 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.544004 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48a5223-q222p\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:24.546043 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.546023 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48a5223-q222p\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:24.566983 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.566950 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8szqb\" (UniqueName: \"kubernetes.io/projected/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-kube-api-access-8szqb\") pod \"maas-api-revoke-sub-d48a5223-q222p\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:24.588465 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.588433 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:24.910579 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:24.910418 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p"] Oct 30 07:00:24.913460 ip-10-0-128-31 kubenswrapper[5001]: W1030 07:00:24.913427 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbd8d2d_ceca_4e83_a933_57e460c78ee6.slice/crio-db10af865d94258bf5f6500fcf2ed1f1ea44ab0014552857a9acbd5ecbf114f3 WatchSource:0}: Error finding container db10af865d94258bf5f6500fcf2ed1f1ea44ab0014552857a9acbd5ecbf114f3: Status 404 returned error can't find the container with id db10af865d94258bf5f6500fcf2ed1f1ea44ab0014552857a9acbd5ecbf114f3 Oct 30 07:00:25.482396 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:25.482284 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" event={"ID":"ffbd8d2d-ceca-4e83-a933-57e460c78ee6","Type":"ContainerStarted","Data":"2a49f8aafb6e3fd3b242b6e63b637ddfe1adb6933c6aa48009182b97f108c059"} Oct 30 07:00:25.482396 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:25.482373 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" event={"ID":"ffbd8d2d-ceca-4e83-a933-57e460c78ee6","Type":"ContainerStarted","Data":"db10af865d94258bf5f6500fcf2ed1f1ea44ab0014552857a9acbd5ecbf114f3"} Oct 30 07:00:25.521654 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:25.521595 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" podStartSLOduration=1.521578346 podStartE2EDuration="1.521578346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:25.520885188 +0000 UTC m=+4039.516842198" watchObservedRunningTime="2026-10-30 07:00:25.521578346 +0000 UTC m=+4039.517535358" Oct 30 07:00:26.117901 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.117865 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v"] Oct 30 07:00:26.118423 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.118404 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" Oct 30 07:00:26.118548 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.118424 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" Oct 30 07:00:26.118548 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.118482 5001 cpu_manager.go:414] "containerMap: removing container" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerName="cleanup" Oct 30 07:00:26.118548 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.118490 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d1c8a0-de78-4cc5-85e5-f538ce6237ba" containerName="cleanup" Oct 30 07:00:26.118716 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.118622 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45c44da-7337-4c2e-8a79-f6e76e853db6" containerName="cleanup" Oct 30 07:00:26.118716 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.118641 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4549b201-684a-49f1-b871-4c0866e27bbb" containerName="istio-proxy" Oct 30 07:00:26.118716 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.118650 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df149d0-8b5f-4273-aad4-11a641230f8d" containerName="cleanup" Oct 30 07:00:26.118716 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.118660 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad31106-b066-48b3-9bc4-a8e442cb719e" containerName="istio-proxy" Oct 30 07:00:26.122224 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.122200 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:26.146147 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.146120 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v"] Oct 30 07:00:26.263618 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.263582 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852a6270-0207-411e-9d32-37903673b768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5b1bfd1-vdd8v\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:26.263939 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.263763 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtzr\" (UniqueName: \"kubernetes.io/projected/852a6270-0207-411e-9d32-37903673b768-kube-api-access-cmtzr\") pod \"maas-api-revoke-sub-c5b1bfd1-vdd8v\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:26.263939 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.263843 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852a6270-0207-411e-9d32-37903673b768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5b1bfd1-vdd8v\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:26.364910 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.364861 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtzr\" (UniqueName: \"kubernetes.io/projected/852a6270-0207-411e-9d32-37903673b768-kube-api-access-cmtzr\") pod \"maas-api-revoke-sub-c5b1bfd1-vdd8v\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:26.365248 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.364932 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852a6270-0207-411e-9d32-37903673b768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5b1bfd1-vdd8v\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:26.365248 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.364966 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852a6270-0207-411e-9d32-37903673b768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5b1bfd1-vdd8v\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:26.365746 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.365724 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852a6270-0207-411e-9d32-37903673b768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5b1bfd1-vdd8v\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:26.367595 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.367567 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852a6270-0207-411e-9d32-37903673b768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5b1bfd1-vdd8v\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:26.388257 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.388192 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtzr\" (UniqueName: \"kubernetes.io/projected/852a6270-0207-411e-9d32-37903673b768-kube-api-access-cmtzr\") pod \"maas-api-revoke-sub-c5b1bfd1-vdd8v\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:26.437468 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.437436 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:26.495716 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.495681 5001 generic.go:356] "Generic (PLEG): container finished" podID="ffbd8d2d-ceca-4e83-a933-57e460c78ee6" containerID="2a49f8aafb6e3fd3b242b6e63b637ddfe1adb6933c6aa48009182b97f108c059" exitCode=0 Oct 30 07:00:26.496115 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.495740 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" event={"ID":"ffbd8d2d-ceca-4e83-a933-57e460c78ee6","Type":"ContainerDied","Data":"2a49f8aafb6e3fd3b242b6e63b637ddfe1adb6933c6aa48009182b97f108c059"} Oct 30 07:00:26.747595 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:26.747562 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v"] Oct 30 07:00:26.750533 ip-10-0-128-31 kubenswrapper[5001]: W1030 07:00:26.750499 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852a6270_0207_411e_9d32_37903673b768.slice/crio-cdb4b655dbeb000ec039f0a2ee35cc735d51acac076c7e22945312efc6c91a8a WatchSource:0}: Error finding container cdb4b655dbeb000ec039f0a2ee35cc735d51acac076c7e22945312efc6c91a8a: Status 404 returned error can't find the container with id cdb4b655dbeb000ec039f0a2ee35cc735d51acac076c7e22945312efc6c91a8a Oct 30 07:00:27.251586 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.251530 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" podUID="4bd74b66-5969-4ae8-bd39-57cb48fbd1e2" containerName="payload-pre-processing" containerID="cri-o://075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8" gracePeriod=25 Oct 30 07:00:27.291138 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.291027 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 07:00:27.376904 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.376874 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b05b248-f62b-466f-b923-311b6d99c48e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b05b248-f62b-466f-b923-311b6d99c48e-plugins-config-volume\") pod \"0b05b248-f62b-466f-b923-311b6d99c48e\" (UID: \"0b05b248-f62b-466f-b923-311b6d99c48e\") " Oct 30 07:00:27.377092 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.376981 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b05b248-f62b-466f-b923-311b6d99c48e-kube-api-access-rcbsn\" (UniqueName: \"kubernetes.io/projected/0b05b248-f62b-466f-b923-311b6d99c48e-kube-api-access-rcbsn\") pod \"0b05b248-f62b-466f-b923-311b6d99c48e\" (UID: \"0b05b248-f62b-466f-b923-311b6d99c48e\") " Oct 30 07:00:27.377300 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.377279 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b05b248-f62b-466f-b923-311b6d99c48e-plugins-config-volume" pod "0b05b248-f62b-466f-b923-311b6d99c48e" (UID: "0b05b248-f62b-466f-b923-311b6d99c48e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:27.377404 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.377390 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b05b248-f62b-466f-b923-311b6d99c48e-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:27.379133 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.379113 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b05b248-f62b-466f-b923-311b6d99c48e-kube-api-access-rcbsn" pod "0b05b248-f62b-466f-b923-311b6d99c48e" (UID: "0b05b248-f62b-466f-b923-311b6d99c48e"). InnerVolumeSpecName "kube-api-access-rcbsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:27.478318 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.478268 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcbsn\" (UniqueName: \"kubernetes.io/projected/0b05b248-f62b-466f-b923-311b6d99c48e-kube-api-access-rcbsn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:27.504723 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.504685 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" event={"ID":"852a6270-0207-411e-9d32-37903673b768","Type":"ContainerStarted","Data":"bf38077c42109426eee26ed024801384e2d50b68173b8c16c1564e0f203c2407"} Oct 30 07:00:27.505206 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.504737 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" event={"ID":"852a6270-0207-411e-9d32-37903673b768","Type":"ContainerStarted","Data":"cdb4b655dbeb000ec039f0a2ee35cc735d51acac076c7e22945312efc6c91a8a"} Oct 30 07:00:27.506075 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.506052 5001 generic.go:356] "Generic (PLEG): container finished" podID="0b05b248-f62b-466f-b923-311b6d99c48e" containerID="550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0" exitCode=0 Oct 30 07:00:27.506193 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.506092 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" Oct 30 07:00:27.506193 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.506135 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" event={"ID":"0b05b248-f62b-466f-b923-311b6d99c48e","Type":"ContainerDied","Data":"550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0"} Oct 30 07:00:27.506193 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.506183 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw" event={"ID":"0b05b248-f62b-466f-b923-311b6d99c48e","Type":"ContainerDied","Data":"06b60a36aa5673a0a0db464fccd5d5bfb9da84ddd74f9ce5360bc64233b1630e"} Oct 30 07:00:27.506336 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.506212 5001 scope.go:122] "RemoveContainer" containerID="550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0" Oct 30 07:00:27.518020 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.518005 5001 scope.go:122] "RemoveContainer" containerID="550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0" Oct 30 07:00:27.518298 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:27.518277 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0\": container with ID starting with 550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0 not found: ID does not exist" containerID="550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0" Oct 30 07:00:27.518427 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.518400 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0"} err="failed to get container status \"550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0\": rpc error: code = NotFound desc = could not find container \"550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0\": container with ID starting with 550c668e9fc88e3f98e2e8bb334b3567b534abcb0332047d3934f7b9258eeee0 not found: ID does not exist" Oct 30 07:00:27.547485 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.547384 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" podStartSLOduration=1.547363805 podStartE2EDuration="1.547363805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:27.542565557 +0000 UTC m=+4041.538522569" watchObservedRunningTime="2026-10-30 07:00:27.547363805 +0000 UTC m=+4041.543320816" Oct 30 07:00:27.578921 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.578780 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw"] Oct 30 07:00:27.583418 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.583333 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-30fbb6-65bd599cc9-n2wxw"] Oct 30 07:00:27.667383 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.667359 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:27.782250 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.782218 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-xks-ca\") pod \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " Oct 30 07:00:27.782487 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.782369 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-openshift-ca\") pod \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " Oct 30 07:00:27.782487 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.782393 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-kube-api-access-8szqb\" (UniqueName: \"kubernetes.io/projected/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-kube-api-access-8szqb\") pod \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\" (UID: \"ffbd8d2d-ceca-4e83-a933-57e460c78ee6\") " Oct 30 07:00:27.782747 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.782724 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-openshift-ca" pod "ffbd8d2d-ceca-4e83-a933-57e460c78ee6" (UID: "ffbd8d2d-ceca-4e83-a933-57e460c78ee6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:27.784658 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.784632 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-xks-ca" pod "ffbd8d2d-ceca-4e83-a933-57e460c78ee6" (UID: "ffbd8d2d-ceca-4e83-a933-57e460c78ee6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:27.784805 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.784777 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-kube-api-access-8szqb" pod "ffbd8d2d-ceca-4e83-a933-57e460c78ee6" (UID: "ffbd8d2d-ceca-4e83-a933-57e460c78ee6"). InnerVolumeSpecName "kube-api-access-8szqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:27.883609 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.883572 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:27.883609 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.883605 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8szqb\" (UniqueName: \"kubernetes.io/projected/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-kube-api-access-8szqb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:27.883827 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:27.883621 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffbd8d2d-ceca-4e83-a933-57e460c78ee6-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.035891 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.035849 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" containerID="cri-o://8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34" gracePeriod=45 Oct 30 07:00:28.118078 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.118039 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b05b248-f62b-466f-b923-311b6d99c48e" path="/var/lib/kubelet/pods/0b05b248-f62b-466f-b923-311b6d99c48e/volumes" Oct 30 07:00:28.305523 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.305496 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 07:00:28.387897 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.387848 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/154bbb27-a34a-4131-a586-e724c31118b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/154bbb27-a34a-4131-a586-e724c31118b6-tls-certs\") pod \"154bbb27-a34a-4131-a586-e724c31118b6\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " Oct 30 07:00:28.388110 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.387950 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-home\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-home\") pod \"154bbb27-a34a-4131-a586-e724c31118b6\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " Oct 30 07:00:28.388110 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.388006 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-model-cache\") pod \"154bbb27-a34a-4131-a586-e724c31118b6\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " Oct 30 07:00:28.388228 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.388108 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-tmp-dir\") pod \"154bbb27-a34a-4131-a586-e724c31118b6\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " Oct 30 07:00:28.388228 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.388162 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154bbb27-a34a-4131-a586-e724c31118b6-kube-api-access-6gdr7\" (UniqueName: \"kubernetes.io/projected/154bbb27-a34a-4131-a586-e724c31118b6-kube-api-access-6gdr7\") pod \"154bbb27-a34a-4131-a586-e724c31118b6\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " Oct 30 07:00:28.388228 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.388209 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-dshm\") pod \"154bbb27-a34a-4131-a586-e724c31118b6\" (UID: \"154bbb27-a34a-4131-a586-e724c31118b6\") " Oct 30 07:00:28.388437 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.388356 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-home" pod "154bbb27-a34a-4131-a586-e724c31118b6" (UID: "154bbb27-a34a-4131-a586-e724c31118b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.388613 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.388562 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-model-cache" pod "154bbb27-a34a-4131-a586-e724c31118b6" (UID: "154bbb27-a34a-4131-a586-e724c31118b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.388759 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.388691 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.388759 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.388710 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.388759 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.388717 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-tmp-dir" pod "154bbb27-a34a-4131-a586-e724c31118b6" (UID: "154bbb27-a34a-4131-a586-e724c31118b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.390923 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.390886 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/154bbb27-a34a-4131-a586-e724c31118b6-tls-certs" pod "154bbb27-a34a-4131-a586-e724c31118b6" (UID: "154bbb27-a34a-4131-a586-e724c31118b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:28.391049 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.391019 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154bbb27-a34a-4131-a586-e724c31118b6-kube-api-access-6gdr7" pod "154bbb27-a34a-4131-a586-e724c31118b6" (UID: "154bbb27-a34a-4131-a586-e724c31118b6"). InnerVolumeSpecName "kube-api-access-6gdr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:28.391299 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.391283 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-dshm" pod "154bbb27-a34a-4131-a586-e724c31118b6" (UID: "154bbb27-a34a-4131-a586-e724c31118b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.489808 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.489726 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gdr7\" (UniqueName: \"kubernetes.io/projected/154bbb27-a34a-4131-a586-e724c31118b6-kube-api-access-6gdr7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.489808 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.489754 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.489808 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.489764 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154bbb27-a34a-4131-a586-e724c31118b6-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.489808 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.489774 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154bbb27-a34a-4131-a586-e724c31118b6-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.517019 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.516981 5001 generic.go:356] "Generic (PLEG): container finished" podID="852a6270-0207-411e-9d32-37903673b768" containerID="bf38077c42109426eee26ed024801384e2d50b68173b8c16c1564e0f203c2407" exitCode=0 Oct 30 07:00:28.517490 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.517058 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" event={"ID":"852a6270-0207-411e-9d32-37903673b768","Type":"ContainerDied","Data":"bf38077c42109426eee26ed024801384e2d50b68173b8c16c1564e0f203c2407"} Oct 30 07:00:28.518407 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.518375 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" podUID="0f4b00a4-8f73-4fa9-9e96-7100bce648a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:28.519889 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.519865 5001 generic.go:356] "Generic (PLEG): container finished" podID="154bbb27-a34a-4131-a586-e724c31118b6" containerID="8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34" exitCode=0 Oct 30 07:00:28.519996 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.519905 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" Oct 30 07:00:28.519996 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.519943 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" event={"ID":"154bbb27-a34a-4131-a586-e724c31118b6","Type":"ContainerDied","Data":"8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34"} Oct 30 07:00:28.519996 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.519982 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" event={"ID":"154bbb27-a34a-4131-a586-e724c31118b6","Type":"ContainerDied","Data":"812b21592bff45da023b09bdb95fa632888aa5d38ba4f43a225aecf9e50bbcec"} Oct 30 07:00:28.520174 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.520002 5001 scope.go:122] "RemoveContainer" containerID="8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34" Oct 30 07:00:28.521556 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.521536 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" Oct 30 07:00:28.521695 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.521555 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48a5223-q222p" event={"ID":"ffbd8d2d-ceca-4e83-a933-57e460c78ee6","Type":"ContainerDied","Data":"db10af865d94258bf5f6500fcf2ed1f1ea44ab0014552857a9acbd5ecbf114f3"} Oct 30 07:00:28.521695 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.521594 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db10af865d94258bf5f6500fcf2ed1f1ea44ab0014552857a9acbd5ecbf114f3" Oct 30 07:00:28.538013 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.537996 5001 scope.go:122] "RemoveContainer" containerID="8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34" Oct 30 07:00:28.538367 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:28.538343 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34\": container with ID starting with 8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34 not found: ID does not exist" containerID="8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34" Oct 30 07:00:28.538456 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.538378 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34"} err="failed to get container status \"8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34\": rpc error: code = NotFound desc = could not find container \"8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34\": container with ID starting with 8a6027821eeb02354237ae638f3b76f852f5b7df91c13f464c2d319c8452da34 not found: ID does not exist" Oct 30 07:00:28.620535 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.620501 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx"] Oct 30 07:00:28.624520 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:28.624485 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx"] Oct 30 07:00:29.290813 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.290764 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-2-w2-aaf467-kserve-cfccb47-qgfvx" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 07:00:29.686505 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.686474 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:29.805548 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.805517 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852a6270-0207-411e-9d32-37903673b768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852a6270-0207-411e-9d32-37903673b768-maas-api-openshift-ca\") pod \"852a6270-0207-411e-9d32-37903673b768\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " Oct 30 07:00:29.805739 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.805675 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852a6270-0207-411e-9d32-37903673b768-kube-api-access-cmtzr\" (UniqueName: \"kubernetes.io/projected/852a6270-0207-411e-9d32-37903673b768-kube-api-access-cmtzr\") pod \"852a6270-0207-411e-9d32-37903673b768\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " Oct 30 07:00:29.805739 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.805702 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/852a6270-0207-411e-9d32-37903673b768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852a6270-0207-411e-9d32-37903673b768-maas-api-xks-ca\") pod \"852a6270-0207-411e-9d32-37903673b768\" (UID: \"852a6270-0207-411e-9d32-37903673b768\") " Oct 30 07:00:29.805965 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.805938 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852a6270-0207-411e-9d32-37903673b768-maas-api-openshift-ca" pod "852a6270-0207-411e-9d32-37903673b768" (UID: "852a6270-0207-411e-9d32-37903673b768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:29.808044 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.808016 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/852a6270-0207-411e-9d32-37903673b768-maas-api-xks-ca" pod "852a6270-0207-411e-9d32-37903673b768" (UID: "852a6270-0207-411e-9d32-37903673b768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:29.808205 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.808156 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852a6270-0207-411e-9d32-37903673b768-kube-api-access-cmtzr" pod "852a6270-0207-411e-9d32-37903673b768" (UID: "852a6270-0207-411e-9d32-37903673b768"). InnerVolumeSpecName "kube-api-access-cmtzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:29.850570 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.850497 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" podUID="c6d8db02-a932-402c-aa37-4e51ac01c609" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:29.906866 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.906831 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmtzr\" (UniqueName: \"kubernetes.io/projected/852a6270-0207-411e-9d32-37903673b768-kube-api-access-cmtzr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:29.906866 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.906863 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852a6270-0207-411e-9d32-37903673b768-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:29.906866 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:29.906872 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852a6270-0207-411e-9d32-37903673b768-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:30.118487 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:30.118456 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154bbb27-a34a-4131-a586-e724c31118b6" path="/var/lib/kubelet/pods/154bbb27-a34a-4131-a586-e724c31118b6/volumes" Oct 30 07:00:30.539286 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:30.539202 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" Oct 30 07:00:30.539286 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:30.539246 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b1bfd1-vdd8v" event={"ID":"852a6270-0207-411e-9d32-37903673b768","Type":"ContainerDied","Data":"cdb4b655dbeb000ec039f0a2ee35cc735d51acac076c7e22945312efc6c91a8a"} Oct 30 07:00:30.539512 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:30.539294 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdb4b655dbeb000ec039f0a2ee35cc735d51acac076c7e22945312efc6c91a8a" Oct 30 07:00:31.450925 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.450876 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" containerID="cri-o://6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185" gracePeriod=45 Oct 30 07:00:31.550381 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.550339 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Oct 30 07:00:31.679921 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.679882 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr"] Oct 30 07:00:31.709043 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.709011 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5"] Oct 30 07:00:31.727554 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.727521 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 07:00:31.828544 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.828507 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tls-certs\" (UniqueName: \"kubernetes.io/secret/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tls-certs\") pod \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " Oct 30 07:00:31.828763 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.828621 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-dshm\") pod \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " Oct 30 07:00:31.828763 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.828694 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-model-cache\") pod \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " Oct 30 07:00:31.828763 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.828726 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-kube-api-access-d7k7v\" (UniqueName: \"kubernetes.io/projected/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-kube-api-access-d7k7v\") pod \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " Oct 30 07:00:31.828950 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.828765 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-home\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-home\") pod \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " Oct 30 07:00:31.828950 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.828811 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tmp-dir\") pod \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\" (UID: \"73e78f9f-36ac-4ec3-a7ce-da47e56efc17\") " Oct 30 07:00:31.829088 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.829058 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-model-cache" pod "73e78f9f-36ac-4ec3-a7ce-da47e56efc17" (UID: "73e78f9f-36ac-4ec3-a7ce-da47e56efc17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.829234 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.829216 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.829326 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.829274 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tmp-dir" pod "73e78f9f-36ac-4ec3-a7ce-da47e56efc17" (UID: "73e78f9f-36ac-4ec3-a7ce-da47e56efc17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.829411 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.829385 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-home" pod "73e78f9f-36ac-4ec3-a7ce-da47e56efc17" (UID: "73e78f9f-36ac-4ec3-a7ce-da47e56efc17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.831267 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.831233 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-dshm" pod "73e78f9f-36ac-4ec3-a7ce-da47e56efc17" (UID: "73e78f9f-36ac-4ec3-a7ce-da47e56efc17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.831436 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.831299 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-kube-api-access-d7k7v" pod "73e78f9f-36ac-4ec3-a7ce-da47e56efc17" (UID: "73e78f9f-36ac-4ec3-a7ce-da47e56efc17"). InnerVolumeSpecName "kube-api-access-d7k7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:31.831632 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.831617 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tls-certs" pod "73e78f9f-36ac-4ec3-a7ce-da47e56efc17" (UID: "73e78f9f-36ac-4ec3-a7ce-da47e56efc17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:31.931105 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.931061 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.931105 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.931104 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.931350 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.931119 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7k7v\" (UniqueName: \"kubernetes.io/projected/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-kube-api-access-d7k7v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.931350 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.931131 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.931350 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:31.931143 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73e78f9f-36ac-4ec3-a7ce-da47e56efc17-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:32.556759 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:32.556665 5001 generic.go:356] "Generic (PLEG): container finished" podID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerID="6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185" exitCode=0 Oct 30 07:00:32.556759 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:32.556728 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" Oct 30 07:00:32.556759 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:32.556746 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" event={"ID":"73e78f9f-36ac-4ec3-a7ce-da47e56efc17","Type":"ContainerDied","Data":"6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185"} Oct 30 07:00:32.557404 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:32.556791 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g" event={"ID":"73e78f9f-36ac-4ec3-a7ce-da47e56efc17","Type":"ContainerDied","Data":"bc2f4d2a1307d6855e1bb7903cbc0810be0b384e37fda6db4a4f0434f11c520e"} Oct 30 07:00:32.557404 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:32.556813 5001 scope.go:122] "RemoveContainer" containerID="6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185" Oct 30 07:00:32.567791 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:32.567771 5001 scope.go:122] "RemoveContainer" containerID="6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185" Oct 30 07:00:32.568156 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:32.568128 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185\": container with ID starting with 6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185 not found: ID does not exist" containerID="6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185" Oct 30 07:00:32.568224 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:32.568168 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185"} err="failed to get container status \"6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185\": rpc error: code = NotFound desc = could not find container \"6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185\": container with ID starting with 6fd93e329942685cc18b8cae1435ab9b22d710bb91f3b4831ab1e4a898463185 not found: ID does not exist" Oct 30 07:00:32.609465 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:32.609437 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g"] Oct 30 07:00:32.620551 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:32.620522 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-2-w3-8d6349-kserve-598b76c7b9-mt99g"] Oct 30 07:00:33.402272 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.402245 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" Oct 30 07:00:33.457963 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.457873 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-kube-api-access-xrcsj\" (UniqueName: \"kubernetes.io/projected/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-kube-api-access-xrcsj\") pod \"e8a8e120-452d-4dbc-8af4-09df0ce99e6d\" (UID: \"e8a8e120-452d-4dbc-8af4-09df0ce99e6d\") " Oct 30 07:00:33.457963 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.457950 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-maas-api-service-ca\") pod \"e8a8e120-452d-4dbc-8af4-09df0ce99e6d\" (UID: \"e8a8e120-452d-4dbc-8af4-09df0ce99e6d\") " Oct 30 07:00:33.458883 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.458849 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-maas-api-service-ca" pod "e8a8e120-452d-4dbc-8af4-09df0ce99e6d" (UID: "e8a8e120-452d-4dbc-8af4-09df0ce99e6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:33.460350 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.460322 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-kube-api-access-xrcsj" pod "e8a8e120-452d-4dbc-8af4-09df0ce99e6d" (UID: "e8a8e120-452d-4dbc-8af4-09df0ce99e6d"). InnerVolumeSpecName "kube-api-access-xrcsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:33.559646 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.559612 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrcsj\" (UniqueName: \"kubernetes.io/projected/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-kube-api-access-xrcsj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:33.559646 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.559643 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a8e120-452d-4dbc-8af4-09df0ce99e6d-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:33.567677 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.567646 5001 generic.go:356] "Generic (PLEG): container finished" podID="e8a8e120-452d-4dbc-8af4-09df0ce99e6d" containerID="2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d" exitCode=137 Oct 30 07:00:33.567863 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.567703 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" Oct 30 07:00:33.567863 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.567724 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" event={"ID":"e8a8e120-452d-4dbc-8af4-09df0ce99e6d","Type":"ContainerDied","Data":"2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d"} Oct 30 07:00:33.567863 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.567778 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc" event={"ID":"e8a8e120-452d-4dbc-8af4-09df0ce99e6d","Type":"ContainerDied","Data":"a68fe7f85461b9b51b24af6dfcc0152b8556144d339430b064c6346acb59acb0"} Oct 30 07:00:33.567863 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.567800 5001 scope.go:122] "RemoveContainer" containerID="2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d" Oct 30 07:00:33.578906 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.578884 5001 scope.go:122] "RemoveContainer" containerID="40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2" Oct 30 07:00:33.591032 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.591011 5001 scope.go:122] "RemoveContainer" containerID="2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d" Oct 30 07:00:33.591411 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:33.591377 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d\": container with ID starting with 2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d not found: ID does not exist" containerID="2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d" Oct 30 07:00:33.591524 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.591420 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d"} err="failed to get container status \"2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d\": rpc error: code = NotFound desc = could not find container \"2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d\": container with ID starting with 2e699172db45f94048f10894f790f1d4815b37c1f61ce3ede130b8c71fc2339d not found: ID does not exist" Oct 30 07:00:33.591524 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.591439 5001 scope.go:122] "RemoveContainer" containerID="40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2" Oct 30 07:00:33.591731 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:33.591707 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2\": container with ID starting with 40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2 not found: ID does not exist" containerID="40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2" Oct 30 07:00:33.591868 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.591742 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2"} err="failed to get container status \"40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2\": rpc error: code = NotFound desc = could not find container \"40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2\": container with ID starting with 40cd4a5ac95be59870261381c9841463fc337eeb606ced8752d8412799e1c7d2 not found: ID does not exist" Oct 30 07:00:33.619127 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.619092 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc"] Oct 30 07:00:33.624881 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:33.624854 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889060-shsqc"] Oct 30 07:00:34.116131 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:34.116100 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" path="/var/lib/kubelet/pods/73e78f9f-36ac-4ec3-a7ce-da47e56efc17/volumes" Oct 30 07:00:34.116633 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:34.116605 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a8e120-452d-4dbc-8af4-09df0ce99e6d" path="/var/lib/kubelet/pods/e8a8e120-452d-4dbc-8af4-09df0ce99e6d/volumes" Oct 30 07:00:36.500998 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:36.500953 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" podUID="4bd74b66-5969-4ae8-bd39-57cb48fbd1e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:37.510353 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.507499 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w"] Oct 30 07:00:37.543176 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.543137 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg"] Oct 30 07:00:37.865596 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.865550 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb"] Oct 30 07:00:37.866246 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866223 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e8a8e120-452d-4dbc-8af4-09df0ce99e6d" containerName="cleanup" Oct 30 07:00:37.866466 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866443 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a8e120-452d-4dbc-8af4-09df0ce99e6d" containerName="cleanup" Oct 30 07:00:37.866590 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866480 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0b05b248-f62b-466f-b923-311b6d99c48e" containerName="payload-pre-processing" Oct 30 07:00:37.866590 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866488 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b05b248-f62b-466f-b923-311b6d99c48e" containerName="payload-pre-processing" Oct 30 07:00:37.866590 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866505 5001 cpu_manager.go:414] "containerMap: removing container" podUID="852a6270-0207-411e-9d32-37903673b768" containerName="revoke-keys" Oct 30 07:00:37.866590 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866513 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852a6270-0207-411e-9d32-37903673b768" containerName="revoke-keys" Oct 30 07:00:37.866590 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866533 5001 cpu_manager.go:414] "containerMap: removing container" podUID="ffbd8d2d-ceca-4e83-a933-57e460c78ee6" containerName="revoke-keys" Oct 30 07:00:37.866590 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866540 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffbd8d2d-ceca-4e83-a933-57e460c78ee6" containerName="revoke-keys" Oct 30 07:00:37.866590 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866560 5001 cpu_manager.go:414] "containerMap: removing container" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" Oct 30 07:00:37.866590 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866567 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" Oct 30 07:00:37.866590 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866576 5001 cpu_manager.go:414] "containerMap: removing container" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" Oct 30 07:00:37.866590 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866582 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" Oct 30 07:00:37.867072 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866722 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffbd8d2d-ceca-4e83-a933-57e460c78ee6" containerName="revoke-keys" Oct 30 07:00:37.867072 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866738 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e78f9f-36ac-4ec3-a7ce-da47e56efc17" containerName="main" Oct 30 07:00:37.867072 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866746 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="154bbb27-a34a-4131-a586-e724c31118b6" containerName="main" Oct 30 07:00:37.867072 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866754 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a8e120-452d-4dbc-8af4-09df0ce99e6d" containerName="cleanup" Oct 30 07:00:37.867072 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866768 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a8e120-452d-4dbc-8af4-09df0ce99e6d" containerName="cleanup" Oct 30 07:00:37.867072 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866779 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b05b248-f62b-466f-b923-311b6d99c48e" containerName="payload-pre-processing" Oct 30 07:00:37.867072 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.866789 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="852a6270-0207-411e-9d32-37903673b768" containerName="revoke-keys" Oct 30 07:00:37.872072 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.872050 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:37.891502 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.891468 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:00:37.891690 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.891506 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:00:37.894036 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.894007 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb"] Oct 30 07:00:37.903708 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.903666 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:00:37.903885 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.903737 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 07:00:37.926473 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.926418 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89e56591-4hzrb\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:37.926736 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.926542 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89e56591-4hzrb\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:37.926736 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:37.926628 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqn92\" (UniqueName: \"kubernetes.io/projected/357d3882-2431-496f-9ddd-7c378d4115e6-kube-api-access-mqn92\") pod \"maas-api-revoke-sub-89e56591-4hzrb\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:38.028085 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:38.028044 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89e56591-4hzrb\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:38.028382 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:38.028148 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqn92\" (UniqueName: \"kubernetes.io/projected/357d3882-2431-496f-9ddd-7c378d4115e6-kube-api-access-mqn92\") pod \"maas-api-revoke-sub-89e56591-4hzrb\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:38.028382 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:38.028275 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89e56591-4hzrb\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:38.028903 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:38.028881 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89e56591-4hzrb\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:38.030991 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:38.030968 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89e56591-4hzrb\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:38.052065 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:38.052037 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqn92\" (UniqueName: \"kubernetes.io/projected/357d3882-2431-496f-9ddd-7c378d4115e6-kube-api-access-mqn92\") pod \"maas-api-revoke-sub-89e56591-4hzrb\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:38.202280 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:38.202193 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:38.276600 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:38.276540 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 07:00:38.518892 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:38.518783 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" podUID="0f4b00a4-8f73-4fa9-9e96-7100bce648a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:38.537154 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:38.537122 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb"] Oct 30 07:00:38.540895 ip-10-0-128-31 kubenswrapper[5001]: W1030 07:00:38.540625 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357d3882_2431_496f_9ddd_7c378d4115e6.slice/crio-6ed42062baa643830b627fe15d988a90c4bc0b4ab744a536d95acca2356f0e37 WatchSource:0}: Error finding container 6ed42062baa643830b627fe15d988a90c4bc0b4ab744a536d95acca2356f0e37: Status 404 returned error can't find the container with id 6ed42062baa643830b627fe15d988a90c4bc0b4ab744a536d95acca2356f0e37 Oct 30 07:00:38.652048 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:38.651998 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" event={"ID":"357d3882-2431-496f-9ddd-7c378d4115e6","Type":"ContainerStarted","Data":"6ed42062baa643830b627fe15d988a90c4bc0b4ab744a536d95acca2356f0e37"} Oct 30 07:00:39.418873 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.418845 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 07:00:39.558319 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.558266 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d8db02-a932-402c-aa37-4e51ac01c609-kube-api-access-s2k45\" (UniqueName: \"kubernetes.io/projected/c6d8db02-a932-402c-aa37-4e51ac01c609-kube-api-access-s2k45\") pod \"c6d8db02-a932-402c-aa37-4e51ac01c609\" (UID: \"c6d8db02-a932-402c-aa37-4e51ac01c609\") " Oct 30 07:00:39.558837 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.558456 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d8db02-a932-402c-aa37-4e51ac01c609-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d8db02-a932-402c-aa37-4e51ac01c609-plugins-config-volume\") pod \"c6d8db02-a932-402c-aa37-4e51ac01c609\" (UID: \"c6d8db02-a932-402c-aa37-4e51ac01c609\") " Oct 30 07:00:39.558901 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.558845 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d8db02-a932-402c-aa37-4e51ac01c609-plugins-config-volume" pod "c6d8db02-a932-402c-aa37-4e51ac01c609" (UID: "c6d8db02-a932-402c-aa37-4e51ac01c609"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:39.560788 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.560764 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d8db02-a932-402c-aa37-4e51ac01c609-kube-api-access-s2k45" pod "c6d8db02-a932-402c-aa37-4e51ac01c609" (UID: "c6d8db02-a932-402c-aa37-4e51ac01c609"). InnerVolumeSpecName "kube-api-access-s2k45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:39.659739 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.659637 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2k45\" (UniqueName: \"kubernetes.io/projected/c6d8db02-a932-402c-aa37-4e51ac01c609-kube-api-access-s2k45\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:39.659739 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.659679 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d8db02-a932-402c-aa37-4e51ac01c609-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:39.663414 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.663386 5001 generic.go:356] "Generic (PLEG): container finished" podID="357d3882-2431-496f-9ddd-7c378d4115e6" containerID="e66de9709f0047ebe61eeb28ef8598b56a924f61c7859d83811e4580b80b2139" exitCode=0 Oct 30 07:00:39.663612 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.663483 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" event={"ID":"357d3882-2431-496f-9ddd-7c378d4115e6","Type":"ContainerDied","Data":"e66de9709f0047ebe61eeb28ef8598b56a924f61c7859d83811e4580b80b2139"} Oct 30 07:00:39.665188 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.665166 5001 generic.go:356] "Generic (PLEG): container finished" podID="c6d8db02-a932-402c-aa37-4e51ac01c609" containerID="168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28" exitCode=0 Oct 30 07:00:39.665323 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.665222 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" Oct 30 07:00:39.665323 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.665235 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" event={"ID":"c6d8db02-a932-402c-aa37-4e51ac01c609","Type":"ContainerDied","Data":"168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28"} Oct 30 07:00:39.665323 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.665292 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn" event={"ID":"c6d8db02-a932-402c-aa37-4e51ac01c609","Type":"ContainerDied","Data":"22a229ea62330dc56d2f94c8627a7b891f954684583e1516c871982307060881"} Oct 30 07:00:39.665505 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.665349 5001 scope.go:122] "RemoveContainer" containerID="168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28" Oct 30 07:00:39.678382 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.678361 5001 scope.go:122] "RemoveContainer" containerID="168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28" Oct 30 07:00:39.678739 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:39.678717 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28\": container with ID starting with 168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28 not found: ID does not exist" containerID="168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28" Oct 30 07:00:39.678790 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.678750 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28"} err="failed to get container status \"168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28\": rpc error: code = NotFound desc = could not find container \"168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28\": container with ID starting with 168bbccbe8024d30b6093ded015da1a6e53167aa4259fa9befded5e3ebd69d28 not found: ID does not exist" Oct 30 07:00:39.739374 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.739336 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn"] Oct 30 07:00:39.747004 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:39.746971 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fa5a69-66fc6646dd-zpmgn"] Oct 30 07:00:40.117550 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:40.117517 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d8db02-a932-402c-aa37-4e51ac01c609" path="/var/lib/kubelet/pods/c6d8db02-a932-402c-aa37-4e51ac01c609/volumes" Oct 30 07:00:40.828878 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:40.828852 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:40.972662 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:40.972570 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-openshift-ca\") pod \"357d3882-2431-496f-9ddd-7c378d4115e6\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " Oct 30 07:00:40.972851 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:40.972665 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357d3882-2431-496f-9ddd-7c378d4115e6-kube-api-access-mqn92\" (UniqueName: \"kubernetes.io/projected/357d3882-2431-496f-9ddd-7c378d4115e6-kube-api-access-mqn92\") pod \"357d3882-2431-496f-9ddd-7c378d4115e6\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " Oct 30 07:00:40.972851 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:40.972813 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-xks-ca\") pod \"357d3882-2431-496f-9ddd-7c378d4115e6\" (UID: \"357d3882-2431-496f-9ddd-7c378d4115e6\") " Oct 30 07:00:40.973066 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:40.973040 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-openshift-ca" pod "357d3882-2431-496f-9ddd-7c378d4115e6" (UID: "357d3882-2431-496f-9ddd-7c378d4115e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:40.975222 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:40.975197 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-xks-ca" pod "357d3882-2431-496f-9ddd-7c378d4115e6" (UID: "357d3882-2431-496f-9ddd-7c378d4115e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:40.975356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:40.975273 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357d3882-2431-496f-9ddd-7c378d4115e6-kube-api-access-mqn92" pod "357d3882-2431-496f-9ddd-7c378d4115e6" (UID: "357d3882-2431-496f-9ddd-7c378d4115e6"). InnerVolumeSpecName "kube-api-access-mqn92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:41.074605 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:41.074564 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:41.074605 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:41.074597 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqn92\" (UniqueName: \"kubernetes.io/projected/357d3882-2431-496f-9ddd-7c378d4115e6-kube-api-access-mqn92\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:41.074605 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:41.074613 5001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357d3882-2431-496f-9ddd-7c378d4115e6-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:41.683923 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:41.683881 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" Oct 30 07:00:41.684116 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:41.683918 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e56591-4hzrb" event={"ID":"357d3882-2431-496f-9ddd-7c378d4115e6","Type":"ContainerDied","Data":"6ed42062baa643830b627fe15d988a90c4bc0b4ab744a536d95acca2356f0e37"} Oct 30 07:00:41.684116 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:41.683973 5001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed42062baa643830b627fe15d988a90c4bc0b4ab744a536d95acca2356f0e37" Oct 30 07:00:42.508223 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:42.508157 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" podUID="154d7c27-437c-4fde-b888-e93026217476" containerName="payload-processing" containerID="cri-o://c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a" gracePeriod=25 Oct 30 07:00:42.543942 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:42.543877 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" podUID="50081a44-3b27-4b9d-8718-3b52afeb8fc4" containerName="payload-pre-processing" containerID="cri-o://6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c" gracePeriod=25 Oct 30 07:00:43.955453 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:43.955417 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 07:00:44.193691 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.193659 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 07:00:44.309076 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.309045 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-plugins-config-volume\") pod \"0f4b00a4-8f73-4fa9-9e96-7100bce648a8\" (UID: \"0f4b00a4-8f73-4fa9-9e96-7100bce648a8\") " Oct 30 07:00:44.309250 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.309116 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-kube-api-access-5trmt\" (UniqueName: \"kubernetes.io/projected/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-kube-api-access-5trmt\") pod \"0f4b00a4-8f73-4fa9-9e96-7100bce648a8\" (UID: \"0f4b00a4-8f73-4fa9-9e96-7100bce648a8\") " Oct 30 07:00:44.309516 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.309493 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-plugins-config-volume" pod "0f4b00a4-8f73-4fa9-9e96-7100bce648a8" (UID: "0f4b00a4-8f73-4fa9-9e96-7100bce648a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:44.311520 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.311487 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-kube-api-access-5trmt" pod "0f4b00a4-8f73-4fa9-9e96-7100bce648a8" (UID: "0f4b00a4-8f73-4fa9-9e96-7100bce648a8"). InnerVolumeSpecName "kube-api-access-5trmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:44.410395 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.410353 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:44.410395 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.410386 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5trmt\" (UniqueName: \"kubernetes.io/projected/0f4b00a4-8f73-4fa9-9e96-7100bce648a8-kube-api-access-5trmt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:44.714032 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.713993 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" podUID="154d7c27-437c-4fde-b888-e93026217476" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:44.714265 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.714240 5001 generic.go:356] "Generic (PLEG): container finished" podID="0f4b00a4-8f73-4fa9-9e96-7100bce648a8" containerID="1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2" exitCode=0 Oct 30 07:00:44.714364 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.714332 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" Oct 30 07:00:44.714364 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.714334 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" event={"ID":"0f4b00a4-8f73-4fa9-9e96-7100bce648a8","Type":"ContainerDied","Data":"1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2"} Oct 30 07:00:44.714491 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.714404 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh" event={"ID":"0f4b00a4-8f73-4fa9-9e96-7100bce648a8","Type":"ContainerDied","Data":"d783d0fe7e07eb8e8a1c1423871ccf2d6daf226af5d40fffebb3d5fd120095ae"} Oct 30 07:00:44.714491 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.714434 5001 scope.go:122] "RemoveContainer" containerID="1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2" Oct 30 07:00:44.727404 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.727383 5001 scope.go:122] "RemoveContainer" containerID="1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2" Oct 30 07:00:44.727692 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:44.727666 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2\": container with ID starting with 1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2 not found: ID does not exist" containerID="1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2" Oct 30 07:00:44.727781 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.727699 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2"} err="failed to get container status \"1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2\": rpc error: code = NotFound desc = could not find container \"1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2\": container with ID starting with 1cebf0ec8bad709c94b93b10210050cdf74ede3be820187ec9077593c59a95a2 not found: ID does not exist" Oct 30 07:00:44.771629 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.771251 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh"] Oct 30 07:00:44.775823 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:44.775795 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aaf467-b7485f4f5-v6pxh"] Oct 30 07:00:45.720580 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:45.720540 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" podUID="50081a44-3b27-4b9d-8718-3b52afeb8fc4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:46.117319 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:46.117263 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4b00a4-8f73-4fa9-9e96-7100bce648a8" path="/var/lib/kubelet/pods/0f4b00a4-8f73-4fa9-9e96-7100bce648a8/volumes" Oct 30 07:00:46.500810 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:46.500729 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" podUID="4bd74b66-5969-4ae8-bd39-57cb48fbd1e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:46.703854 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:46.703795 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" containerID="cri-o://9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da" gracePeriod=45 Oct 30 07:00:46.730136 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:46.730088 5001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" containerID="cri-o://a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d" gracePeriod=45 Oct 30 07:00:46.748207 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:46.748168 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused" Oct 30 07:00:46.997193 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:46.997166 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 07:00:47.032853 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.032821 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 07:00:47.137829 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.137798 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-tmp-dir\") pod \"7a117db9-0054-4f46-a27d-c3861be4ca0e\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " Oct 30 07:00:47.138003 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.137835 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-model-cache\") pod \"7a117db9-0054-4f46-a27d-c3861be4ca0e\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " Oct 30 07:00:47.138003 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.137863 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96382c2d-6131-412c-96fb-42a678998e1d-kube-api-access-hpxhq\" (UniqueName: \"kubernetes.io/projected/96382c2d-6131-412c-96fb-42a678998e1d-kube-api-access-hpxhq\") pod \"96382c2d-6131-412c-96fb-42a678998e1d\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " Oct 30 07:00:47.138003 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.137907 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a117db9-0054-4f46-a27d-c3861be4ca0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a117db9-0054-4f46-a27d-c3861be4ca0e-tls-certs\") pod \"7a117db9-0054-4f46-a27d-c3861be4ca0e\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " Oct 30 07:00:47.138003 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.137946 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-dshm\") pod \"96382c2d-6131-412c-96fb-42a678998e1d\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " Oct 30 07:00:47.138003 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.137978 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-home\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-home\") pod \"96382c2d-6131-412c-96fb-42a678998e1d\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " Oct 30 07:00:47.138332 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.138161 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-tmp-dir" pod "7a117db9-0054-4f46-a27d-c3861be4ca0e" (UID: "7a117db9-0054-4f46-a27d-c3861be4ca0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.138332 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.138168 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-model-cache" pod "7a117db9-0054-4f46-a27d-c3861be4ca0e" (UID: "7a117db9-0054-4f46-a27d-c3861be4ca0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.138332 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.138228 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-dshm\") pod \"7a117db9-0054-4f46-a27d-c3861be4ca0e\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " Oct 30 07:00:47.138332 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.138229 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-home" pod "96382c2d-6131-412c-96fb-42a678998e1d" (UID: "96382c2d-6131-412c-96fb-42a678998e1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.138332 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.138295 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96382c2d-6131-412c-96fb-42a678998e1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/96382c2d-6131-412c-96fb-42a678998e1d-tls-certs\") pod \"96382c2d-6131-412c-96fb-42a678998e1d\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " Oct 30 07:00:47.138631 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.138346 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-tmp-dir\") pod \"96382c2d-6131-412c-96fb-42a678998e1d\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " Oct 30 07:00:47.138631 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.138405 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-model-cache\") pod \"96382c2d-6131-412c-96fb-42a678998e1d\" (UID: \"96382c2d-6131-412c-96fb-42a678998e1d\") " Oct 30 07:00:47.138631 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.138466 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-home\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-home\") pod \"7a117db9-0054-4f46-a27d-c3861be4ca0e\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " Oct 30 07:00:47.138631 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.138495 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a117db9-0054-4f46-a27d-c3861be4ca0e-kube-api-access-qzp7m\" (UniqueName: \"kubernetes.io/projected/7a117db9-0054-4f46-a27d-c3861be4ca0e-kube-api-access-qzp7m\") pod \"7a117db9-0054-4f46-a27d-c3861be4ca0e\" (UID: \"7a117db9-0054-4f46-a27d-c3861be4ca0e\") " Oct 30 07:00:47.138868 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.138789 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-model-cache" pod "96382c2d-6131-412c-96fb-42a678998e1d" (UID: "96382c2d-6131-412c-96fb-42a678998e1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.139271 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.138957 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-home" pod "7a117db9-0054-4f46-a27d-c3861be4ca0e" (UID: "7a117db9-0054-4f46-a27d-c3861be4ca0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.139271 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.139154 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.139271 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.139178 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.139271 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.139191 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.139271 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.139203 5001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.139271 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.139215 5001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.139271 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.139228 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-tmp-dir" pod "96382c2d-6131-412c-96fb-42a678998e1d" (UID: "96382c2d-6131-412c-96fb-42a678998e1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.140740 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.140707 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a117db9-0054-4f46-a27d-c3861be4ca0e-tls-certs" pod "7a117db9-0054-4f46-a27d-c3861be4ca0e" (UID: "7a117db9-0054-4f46-a27d-c3861be4ca0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:47.140878 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.140843 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96382c2d-6131-412c-96fb-42a678998e1d-tls-certs" pod "96382c2d-6131-412c-96fb-42a678998e1d" (UID: "96382c2d-6131-412c-96fb-42a678998e1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:47.140979 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.140953 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-dshm" pod "96382c2d-6131-412c-96fb-42a678998e1d" (UID: "96382c2d-6131-412c-96fb-42a678998e1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.141044 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.140997 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96382c2d-6131-412c-96fb-42a678998e1d-kube-api-access-hpxhq" pod "96382c2d-6131-412c-96fb-42a678998e1d" (UID: "96382c2d-6131-412c-96fb-42a678998e1d"). InnerVolumeSpecName "kube-api-access-hpxhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:47.141855 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.141833 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-dshm" pod "7a117db9-0054-4f46-a27d-c3861be4ca0e" (UID: "7a117db9-0054-4f46-a27d-c3861be4ca0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.142157 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.142143 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a117db9-0054-4f46-a27d-c3861be4ca0e-kube-api-access-qzp7m" pod "7a117db9-0054-4f46-a27d-c3861be4ca0e" (UID: "7a117db9-0054-4f46-a27d-c3861be4ca0e"). InnerVolumeSpecName "kube-api-access-qzp7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:47.240410 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.240364 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a117db9-0054-4f46-a27d-c3861be4ca0e-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.240410 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.240405 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.240410 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.240415 5001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a117db9-0054-4f46-a27d-c3861be4ca0e-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.240661 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.240423 5001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96382c2d-6131-412c-96fb-42a678998e1d-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.240661 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.240431 5001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96382c2d-6131-412c-96fb-42a678998e1d-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.240661 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.240439 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzp7m\" (UniqueName: \"kubernetes.io/projected/7a117db9-0054-4f46-a27d-c3861be4ca0e-kube-api-access-qzp7m\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.240661 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.240448 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpxhq\" (UniqueName: \"kubernetes.io/projected/96382c2d-6131-412c-96fb-42a678998e1d-kube-api-access-hpxhq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.496807 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.496781 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 07:00:47.645501 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.645467 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-kube-api-access-jc55x\" (UniqueName: \"kubernetes.io/projected/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-kube-api-access-jc55x\") pod \"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2\" (UID: \"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2\") " Oct 30 07:00:47.645701 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.645582 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-plugins-config-volume\") pod \"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2\" (UID: \"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2\") " Oct 30 07:00:47.646025 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.645999 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-plugins-config-volume" pod "4bd74b66-5969-4ae8-bd39-57cb48fbd1e2" (UID: "4bd74b66-5969-4ae8-bd39-57cb48fbd1e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:47.647952 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.647922 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-kube-api-access-jc55x" pod "4bd74b66-5969-4ae8-bd39-57cb48fbd1e2" (UID: "4bd74b66-5969-4ae8-bd39-57cb48fbd1e2"). InnerVolumeSpecName "kube-api-access-jc55x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:47.746446 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.746417 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc55x\" (UniqueName: \"kubernetes.io/projected/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-kube-api-access-jc55x\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.746446 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.746443 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.746922 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.746892 5001 generic.go:356] "Generic (PLEG): container finished" podID="4bd74b66-5969-4ae8-bd39-57cb48fbd1e2" containerID="075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8" exitCode=0 Oct 30 07:00:47.746973 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.746941 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" Oct 30 07:00:47.747023 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.746989 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" event={"ID":"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2","Type":"ContainerDied","Data":"075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8"} Oct 30 07:00:47.747079 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.747052 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g" event={"ID":"4bd74b66-5969-4ae8-bd39-57cb48fbd1e2","Type":"ContainerDied","Data":"6896a7747b01170fe29bc1a5079dc21ca3413f3d1ddca0e62e332dccac12b831"} Oct 30 07:00:47.747138 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.747084 5001 scope.go:122] "RemoveContainer" containerID="075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8" Oct 30 07:00:47.748941 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.748914 5001 generic.go:356] "Generic (PLEG): container finished" podID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerID="9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da" exitCode=0 Oct 30 07:00:47.749071 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.748964 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" Oct 30 07:00:47.749071 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.748999 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" event={"ID":"7a117db9-0054-4f46-a27d-c3861be4ca0e","Type":"ContainerDied","Data":"9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da"} Oct 30 07:00:47.749071 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.749046 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr" event={"ID":"7a117db9-0054-4f46-a27d-c3861be4ca0e","Type":"ContainerDied","Data":"fe511eb97f82bd2b9f8e15b0e73645c789a794fb9b56ab0057d6ce315896eaf9"} Oct 30 07:00:47.750639 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.750618 5001 generic.go:356] "Generic (PLEG): container finished" podID="96382c2d-6131-412c-96fb-42a678998e1d" containerID="a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d" exitCode=0 Oct 30 07:00:47.750756 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.750640 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" event={"ID":"96382c2d-6131-412c-96fb-42a678998e1d","Type":"ContainerDied","Data":"a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d"} Oct 30 07:00:47.750756 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.750668 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" event={"ID":"96382c2d-6131-412c-96fb-42a678998e1d","Type":"ContainerDied","Data":"96b4100baa361d9193a98195acf4271df1b93a9a7424124bacc7d9b0acde244f"} Oct 30 07:00:47.750756 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.750682 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5" Oct 30 07:00:47.760651 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.760631 5001 scope.go:122] "RemoveContainer" containerID="075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8" Oct 30 07:00:47.760958 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:47.760933 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8\": container with ID starting with 075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8 not found: ID does not exist" containerID="075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8" Oct 30 07:00:47.761069 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.760969 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8"} err="failed to get container status \"075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8\": rpc error: code = NotFound desc = could not find container \"075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8\": container with ID starting with 075abe2e0f51f27d65915f98371db9aa29ea7a46ee43aa19a19bac6cc499e5a8 not found: ID does not exist" Oct 30 07:00:47.761069 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.760992 5001 scope.go:122] "RemoveContainer" containerID="9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da" Oct 30 07:00:47.771700 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.771682 5001 scope.go:122] "RemoveContainer" containerID="9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da" Oct 30 07:00:47.772013 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:47.771986 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da\": container with ID starting with 9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da not found: ID does not exist" containerID="9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da" Oct 30 07:00:47.772109 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.772015 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da"} err="failed to get container status \"9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da\": rpc error: code = NotFound desc = could not find container \"9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da\": container with ID starting with 9aa69dd1df35ed2c575f4395c6b367457488ea32ee0377847d64e170a957e9da not found: ID does not exist" Oct 30 07:00:47.772109 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.772033 5001 scope.go:122] "RemoveContainer" containerID="a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d" Oct 30 07:00:47.814158 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.814138 5001 scope.go:122] "RemoveContainer" containerID="a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d" Oct 30 07:00:47.814548 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:00:47.814515 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d\": container with ID starting with a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d not found: ID does not exist" containerID="a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d" Oct 30 07:00:47.814636 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.814559 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d"} err="failed to get container status \"a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d\": rpc error: code = NotFound desc = could not find container \"a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d\": container with ID starting with a34c0912a4b6097b60472b3cf204947291ec482f4ae964bd56c55ab6acf41a8d not found: ID does not exist" Oct 30 07:00:47.827154 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.827123 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g"] Oct 30 07:00:47.830805 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.830780 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8d6349-796776cf47-c7m7g"] Oct 30 07:00:47.864968 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.864932 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5"] Oct 30 07:00:47.869267 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.869241 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/simulator-w1-8d44dd-kserve-6f776df956-rt9z5"] Oct 30 07:00:47.901616 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.901587 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr"] Oct 30 07:00:47.905568 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:47.905545 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-2-w1-8d44dd-kserve-6b4cdf4f47-872hr"] Oct 30 07:00:48.115705 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:48.115669 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd74b66-5969-4ae8-bd39-57cb48fbd1e2" path="/var/lib/kubelet/pods/4bd74b66-5969-4ae8-bd39-57cb48fbd1e2/volumes" Oct 30 07:00:48.116124 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:48.116108 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" path="/var/lib/kubelet/pods/7a117db9-0054-4f46-a27d-c3861be4ca0e/volumes" Oct 30 07:00:48.116507 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:48.116494 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96382c2d-6131-412c-96fb-42a678998e1d" path="/var/lib/kubelet/pods/96382c2d-6131-412c-96fb-42a678998e1d/volumes" Oct 30 07:00:54.714261 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:54.714226 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" podUID="154d7c27-437c-4fde-b888-e93026217476" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:55.721123 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:55.721084 5001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" podUID="50081a44-3b27-4b9d-8718-3b52afeb8fc4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:57.568597 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:00:57.568557 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-66g66"] Oct 30 07:01:01.850208 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:01.850176 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4cbf4fd4-7wnrx_383e8617-b682-40d9-9ad6-8f5be7ba56d2/authorino/0.log" Oct 30 07:01:02.072273 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.072242 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9s8vq_937866c0-656c-45fb-bec7-b19124c11c4e/manager/0.log" Oct 30 07:01:02.271819 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.271783 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-g5mql_f611b837-f97a-4ff0-bc4c-3119f5a8d9ca/manager/0.log" Oct 30 07:01:02.794142 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.794066 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hbpn7_ba380ad4-2195-4cfc-b197-d1568adc5140/manager/1.log" Oct 30 07:01:02.828570 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.828539 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 07:01:02.840844 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.840818 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 07:01:02.880897 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.880864 5001 generic.go:356] "Generic (PLEG): container finished" podID="50081a44-3b27-4b9d-8718-3b52afeb8fc4" containerID="6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c" exitCode=0 Oct 30 07:01:02.881413 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.880916 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" Oct 30 07:01:02.881413 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.880945 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" event={"ID":"50081a44-3b27-4b9d-8718-3b52afeb8fc4","Type":"ContainerDied","Data":"6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c"} Oct 30 07:01:02.881413 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.880998 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg" event={"ID":"50081a44-3b27-4b9d-8718-3b52afeb8fc4","Type":"ContainerDied","Data":"a6ba3d5fcb944df0b389761687e957498a93f36384c0680d42cb0089532803d3"} Oct 30 07:01:02.881413 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.881027 5001 scope.go:122] "RemoveContainer" containerID="6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c" Oct 30 07:01:02.882823 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.882799 5001 generic.go:356] "Generic (PLEG): container finished" podID="154d7c27-437c-4fde-b888-e93026217476" containerID="c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a" exitCode=0 Oct 30 07:01:02.882946 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.882871 5001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" Oct 30 07:01:02.882946 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.882885 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" event={"ID":"154d7c27-437c-4fde-b888-e93026217476","Type":"ContainerDied","Data":"c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a"} Oct 30 07:01:02.882946 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.882919 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w" event={"ID":"154d7c27-437c-4fde-b888-e93026217476","Type":"ContainerDied","Data":"52eb82c3249c441befeba01e57762241719872ab36c16933b5f58d60154cbae0"} Oct 30 07:01:02.892089 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.892059 5001 scope.go:122] "RemoveContainer" containerID="6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c" Oct 30 07:01:02.892557 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:01:02.892534 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c\": container with ID starting with 6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c not found: ID does not exist" containerID="6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c" Oct 30 07:01:02.892688 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.892568 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c"} err="failed to get container status \"6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c\": rpc error: code = NotFound desc = could not find container \"6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c\": container with ID starting with 6c0737d907264708cc8ea8c09a662e27b8303f00497770c45f45dd760885c82c not found: ID does not exist" Oct 30 07:01:02.892688 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.892586 5001 scope.go:122] "RemoveContainer" containerID="c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a" Oct 30 07:01:02.906663 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.906632 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154d7c27-437c-4fde-b888-e93026217476-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154d7c27-437c-4fde-b888-e93026217476-plugins-config-volume\") pod \"154d7c27-437c-4fde-b888-e93026217476\" (UID: \"154d7c27-437c-4fde-b888-e93026217476\") " Oct 30 07:01:02.906802 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.906760 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50081a44-3b27-4b9d-8718-3b52afeb8fc4-kube-api-access-l8lfn\" (UniqueName: \"kubernetes.io/projected/50081a44-3b27-4b9d-8718-3b52afeb8fc4-kube-api-access-l8lfn\") pod \"50081a44-3b27-4b9d-8718-3b52afeb8fc4\" (UID: \"50081a44-3b27-4b9d-8718-3b52afeb8fc4\") " Oct 30 07:01:02.906887 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.906824 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50081a44-3b27-4b9d-8718-3b52afeb8fc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50081a44-3b27-4b9d-8718-3b52afeb8fc4-plugins-config-volume\") pod \"50081a44-3b27-4b9d-8718-3b52afeb8fc4\" (UID: \"50081a44-3b27-4b9d-8718-3b52afeb8fc4\") " Oct 30 07:01:02.906887 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.906858 5001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154d7c27-437c-4fde-b888-e93026217476-kube-api-access-czrz9\" (UniqueName: \"kubernetes.io/projected/154d7c27-437c-4fde-b888-e93026217476-kube-api-access-czrz9\") pod \"154d7c27-437c-4fde-b888-e93026217476\" (UID: \"154d7c27-437c-4fde-b888-e93026217476\") " Oct 30 07:01:02.907085 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.907039 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154d7c27-437c-4fde-b888-e93026217476-plugins-config-volume" pod "154d7c27-437c-4fde-b888-e93026217476" (UID: "154d7c27-437c-4fde-b888-e93026217476"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:01:02.907865 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.907833 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50081a44-3b27-4b9d-8718-3b52afeb8fc4-plugins-config-volume" pod "50081a44-3b27-4b9d-8718-3b52afeb8fc4" (UID: "50081a44-3b27-4b9d-8718-3b52afeb8fc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:01:02.907995 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.907922 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/154d7c27-437c-4fde-b888-e93026217476-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:01:02.907995 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.907948 5001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50081a44-3b27-4b9d-8718-3b52afeb8fc4-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:01:02.908218 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.908188 5001 scope.go:122] "RemoveContainer" containerID="c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a" Oct 30 07:01:02.909005 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:01:02.908974 5001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a\": container with ID starting with c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a not found: ID does not exist" containerID="c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a" Oct 30 07:01:02.909156 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.909017 5001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a"} err="failed to get container status \"c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a\": rpc error: code = NotFound desc = could not find container \"c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a\": container with ID starting with c5a624587e7c20d205ead5fcaabf4d8a30964dc327e8e3ddc243e9091529fe5a not found: ID does not exist" Oct 30 07:01:02.910233 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.910206 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154d7c27-437c-4fde-b888-e93026217476-kube-api-access-czrz9" pod "154d7c27-437c-4fde-b888-e93026217476" (UID: "154d7c27-437c-4fde-b888-e93026217476"). InnerVolumeSpecName "kube-api-access-czrz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:01:02.910845 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:02.910821 5001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50081a44-3b27-4b9d-8718-3b52afeb8fc4-kube-api-access-l8lfn" pod "50081a44-3b27-4b9d-8718-3b52afeb8fc4" (UID: "50081a44-3b27-4b9d-8718-3b52afeb8fc4"). InnerVolumeSpecName "kube-api-access-l8lfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:01:03.008628 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:03.008590 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8lfn\" (UniqueName: \"kubernetes.io/projected/50081a44-3b27-4b9d-8718-3b52afeb8fc4-kube-api-access-l8lfn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:01:03.008628 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:03.008626 5001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czrz9\" (UniqueName: \"kubernetes.io/projected/154d7c27-437c-4fde-b888-e93026217476-kube-api-access-czrz9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\"" Oct 30 07:01:03.021653 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:03.021629 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-66g66_3ed04332-3bea-4244-9a83-4976f78ee337/limitador/0.log" Oct 30 07:01:03.224966 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:03.224930 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-thdmg_8f6114c0-230e-40ef-9cb5-60f49d0f4825/manager/0.log" Oct 30 07:01:03.226469 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:03.226446 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg"] Oct 30 07:01:03.237589 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:03.237548 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8d44dd-5994685567-pfkgg"] Oct 30 07:01:03.261360 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:03.261319 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w"] Oct 30 07:01:03.265667 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:03.265638 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8d44dd-f699bfc76-46t7w"] Oct 30 07:01:04.102896 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:04.102857 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hbpn7_ba380ad4-2195-4cfc-b197-d1568adc5140/manager/0.log" Oct 30 07:01:04.115727 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:04.115692 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154d7c27-437c-4fde-b888-e93026217476" path="/var/lib/kubelet/pods/154d7c27-437c-4fde-b888-e93026217476/volumes" Oct 30 07:01:04.116086 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:04.116072 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50081a44-3b27-4b9d-8718-3b52afeb8fc4" path="/var/lib/kubelet/pods/50081a44-3b27-4b9d-8718-3b52afeb8fc4/volumes" Oct 30 07:01:07.476398 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:07.476364 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4cbf4fd4-7wnrx_383e8617-b682-40d9-9ad6-8f5be7ba56d2/authorino/0.log" Oct 30 07:01:08.265426 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:01:08.265388 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 07:01:12.944234 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:12.944201 5001 scope.go:122] "RemoveContainer" containerID="ac2a932c98783917e2cd731f365d9254a9968e28efb4b64a7b95522ba2c9aa39" Oct 30 07:01:21.814772 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:21.814696 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-030c3ea2-vdbkg_ce4cbf99-80b2-45f9-bfec-bbc0fc224a28/revoke-keys/0.log" Oct 30 07:01:22.012353 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:22.012289 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-56624813-frzbs_a8b0dfc3-0c50-4813-a6c1-6f613b713c30/revoke-keys/0.log" Oct 30 07:01:22.390730 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:22.390699 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6d9390f6-cbhrd_020d4ae7-95d3-4808-ab77-27a92b540d20/revoke-keys/0.log" Oct 30 07:01:22.582876 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:22.582847 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-89e56591-4hzrb_357d3882-2431-496f-9ddd-7c378d4115e6/revoke-keys/0.log" Oct 30 07:01:22.760188 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:22.760108 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b51b12dd-ntxhj_b0a08015-0f9b-43ac-8777-2456a1d806f0/revoke-keys/0.log" Oct 30 07:01:22.949342 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:22.949298 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c011337f-b7gkx_3364fda6-e33a-4ada-abfa-84cba1506d16/revoke-keys/0.log" Oct 30 07:01:23.150089 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:23.150060 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c5b1bfd1-vdd8v_852a6270-0207-411e-9d32-37903673b768/revoke-keys/0.log" Oct 30 07:01:23.338148 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:23.338116 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d180ea71-8nm2n_08438c69-53b5-4b5e-b614-eacbcf3d8a3d/revoke-keys/0.log" Oct 30 07:01:23.539188 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:23.539098 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2f6e7fc-dnk5d_07561896-3599-4430-adcf-02bff0a0b23b/revoke-keys/0.log" Oct 30 07:01:23.735171 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:23.735132 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d48a5223-q222p_ffbd8d2d-ceca-4e83-a933-57e460c78ee6/revoke-keys/0.log" Oct 30 07:01:24.044346 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:24.044294 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg"] Oct 30 07:01:24.055816 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:24.055784 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030c3ea2-vdbkg"] Oct 30 07:01:24.116342 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:24.116286 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4cbf99-80b2-45f9-bfec-bbc0fc224a28" path="/var/lib/kubelet/pods/ce4cbf99-80b2-45f9-bfec-bbc0fc224a28/volumes" Oct 30 07:01:24.302330 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:24.302225 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ffa253f3-wshcx_cc748569-b462-43b0-b82b-b1adea0b0964/revoke-keys/0.log" Oct 30 07:01:26.131865 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:26.131834 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4cbf4fd4-7wnrx_383e8617-b682-40d9-9ad6-8f5be7ba56d2/authorino/0.log" Oct 30 07:01:26.375155 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:26.375124 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9s8vq_937866c0-656c-45fb-bec7-b19124c11c4e/manager/0.log" Oct 30 07:01:26.592682 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:26.592600 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-g5mql_f611b837-f97a-4ff0-bc4c-3119f5a8d9ca/manager/0.log" Oct 30 07:01:27.097191 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:27.097136 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hbpn7_ba380ad4-2195-4cfc-b197-d1568adc5140/manager/1.log" Oct 30 07:01:27.357061 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:27.357030 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-66g66_3ed04332-3bea-4244-9a83-4976f78ee337/limitador/0.log" Oct 30 07:01:27.555146 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:27.555111 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-thdmg_8f6114c0-230e-40ef-9cb5-60f49d0f4825/manager/0.log" Oct 30 07:01:28.450850 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:28.450806 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hbpn7_ba380ad4-2195-4cfc-b197-d1568adc5140/manager/0.log" Oct 30 07:01:29.076563 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:29.076524 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t_cea443fd-b42f-4897-9731-8354eb3ca6db/istio-proxy/0.log" Oct 30 07:01:29.275672 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:29.275644 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-scslz_e1d1dda1-333e-4206-9933-ffe9acdc95c7/discovery/0.log" Oct 30 07:01:29.897574 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:29.897538 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-5nt9s_aed4e7ce-ff6a-4e62-92da-33fc1092c705/istio-proxy/0.log" Oct 30 07:01:30.318040 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:30.317941 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-s6zsz_ff80c97c-2e4d-402f-88a3-4168cc366e05/istio-proxy/0.log" Oct 30 07:01:30.528722 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:30.528694 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7_7a158a21-1de7-4e61-bfda-d255713a8655/istio-proxy/0.log" Oct 30 07:01:31.162025 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:31.161981 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b994dfc85-2v9ww_84346da8-55f2-4344-b7d0-dbfb2996f2e7/router/0.log" Oct 30 07:01:32.942527 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:32.942490 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-5vzlj_efd5ce32-5bd7-4589-8f5b-30f580257a13/main/0.log" Oct 30 07:01:33.128067 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:33.128037 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-hmvkl_cc5f3ec1-3f05-40ba-a14c-b6ddc6a00db0/main/0.log" Oct 30 07:01:36.039617 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:36.039578 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n"] Oct 30 07:01:36.050913 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:36.050876 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d180ea71-8nm2n"] Oct 30 07:01:36.116275 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:36.116221 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08438c69-53b5-4b5e-b614-eacbcf3d8a3d" path="/var/lib/kubelet/pods/08438c69-53b5-4b5e-b614-eacbcf3d8a3d/volumes" Oct 30 07:01:38.269574 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:01:38.269538 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 07:01:47.045234 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:47.045198 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs"] Oct 30 07:01:47.049887 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:47.049858 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56624813-frzbs"] Oct 30 07:01:48.116368 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:48.116332 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b0dfc3-0c50-4813-a6c1-6f613b713c30" path="/var/lib/kubelet/pods/a8b0dfc3-0c50-4813-a6c1-6f613b713c30/volumes" Oct 30 07:01:49.187197 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:49.187164 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bnv8q_d44df296-4186-4c40-81de-1ed06c15955e/global-pull-secret-syncer/0.log" Oct 30 07:01:49.746800 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:49.746774 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-2626d_c7e5b861-eb47-4d8a-8974-90a2368489ed/connection-checker/0.log" Oct 30 07:01:49.878160 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:49.878131 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-74dvn_02b629c0-be01-461a-88b8-85a69dfad05b/connection-checker/0.log" Oct 30 07:01:50.081026 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:50.080932 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-j7pwr_8cd01f41-97de-437c-aacf-29cb3668570b/connection-checker/0.log" Oct 30 07:01:50.360804 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:50.360776 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cq5xz_3b6113ea-d183-4a72-a574-80c03c8f5352/konnectivity-agent/0.log" Oct 30 07:01:50.678083 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:50.678000 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-31.ec2.internal_ed322e80a89b2bbeafbf0f88fe9597d6/haproxy/0.log" Oct 30 07:01:57.877018 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:57.876980 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4cbf4fd4-7wnrx_383e8617-b682-40d9-9ad6-8f5be7ba56d2/authorino/0.log" Oct 30 07:01:58.035530 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:58.035495 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx"] Oct 30 07:01:58.047252 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:58.047224 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffa253f3-wshcx"] Oct 30 07:01:58.117113 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:58.117077 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc748569-b462-43b0-b82b-b1adea0b0964" path="/var/lib/kubelet/pods/cc748569-b462-43b0-b82b-b1adea0b0964/volumes" Oct 30 07:01:58.141285 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:58.141210 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9s8vq_937866c0-656c-45fb-bec7-b19124c11c4e/manager/0.log" Oct 30 07:01:58.266416 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:58.266390 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-g5mql_f611b837-f97a-4ff0-bc4c-3119f5a8d9ca/manager/0.log" Oct 30 07:01:58.819505 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:58.819464 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hbpn7_ba380ad4-2195-4cfc-b197-d1568adc5140/manager/1.log" Oct 30 07:01:59.891768 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.891721 5001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj"] Oct 30 07:01:59.892218 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892182 5001 cpu_manager.go:414] "containerMap: removing container" podUID="c6d8db02-a932-402c-aa37-4e51ac01c609" containerName="payload-processing" Oct 30 07:01:59.892218 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892201 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d8db02-a932-402c-aa37-4e51ac01c609" containerName="payload-processing" Oct 30 07:01:59.892356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892220 5001 cpu_manager.go:414] "containerMap: removing container" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" Oct 30 07:01:59.892356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892231 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" Oct 30 07:01:59.892356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892240 5001 cpu_manager.go:414] "containerMap: removing container" podUID="4bd74b66-5969-4ae8-bd39-57cb48fbd1e2" containerName="payload-pre-processing" Oct 30 07:01:59.892356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892267 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd74b66-5969-4ae8-bd39-57cb48fbd1e2" containerName="payload-pre-processing" Oct 30 07:01:59.892356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892279 5001 cpu_manager.go:414] "containerMap: removing container" podUID="50081a44-3b27-4b9d-8718-3b52afeb8fc4" containerName="payload-pre-processing" Oct 30 07:01:59.892356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892286 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50081a44-3b27-4b9d-8718-3b52afeb8fc4" containerName="payload-pre-processing" Oct 30 07:01:59.892356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892318 5001 cpu_manager.go:414] "containerMap: removing container" podUID="e8a8e120-452d-4dbc-8af4-09df0ce99e6d" containerName="cleanup" Oct 30 07:01:59.892356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892325 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a8e120-452d-4dbc-8af4-09df0ce99e6d" containerName="cleanup" Oct 30 07:01:59.892356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892341 5001 cpu_manager.go:414] "containerMap: removing container" podUID="154d7c27-437c-4fde-b888-e93026217476" containerName="payload-processing" Oct 30 07:01:59.892356 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892349 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154d7c27-437c-4fde-b888-e93026217476" containerName="payload-processing" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892362 5001 cpu_manager.go:414] "containerMap: removing container" podUID="357d3882-2431-496f-9ddd-7c378d4115e6" containerName="revoke-keys" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892370 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357d3882-2431-496f-9ddd-7c378d4115e6" containerName="revoke-keys" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892384 5001 cpu_manager.go:414] "containerMap: removing container" podUID="0f4b00a4-8f73-4fa9-9e96-7100bce648a8" containerName="payload-pre-processing" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892389 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4b00a4-8f73-4fa9-9e96-7100bce648a8" containerName="payload-pre-processing" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892400 5001 cpu_manager.go:414] "containerMap: removing container" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892404 5001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892490 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4b00a4-8f73-4fa9-9e96-7100bce648a8" containerName="payload-pre-processing" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892515 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="96382c2d-6131-412c-96fb-42a678998e1d" containerName="main" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892523 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d8db02-a932-402c-aa37-4e51ac01c609" containerName="payload-processing" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892530 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a117db9-0054-4f46-a27d-c3861be4ca0e" containerName="main" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892535 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd74b66-5969-4ae8-bd39-57cb48fbd1e2" containerName="payload-pre-processing" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892540 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="50081a44-3b27-4b9d-8718-3b52afeb8fc4" containerName="payload-pre-processing" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892547 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="154d7c27-437c-4fde-b888-e93026217476" containerName="payload-processing" Oct 30 07:01:59.893111 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.892553 5001 memory_manager.go:365] "RemoveStaleState removing state" podUID="357d3882-2431-496f-9ddd-7c378d4115e6" containerName="revoke-keys" Oct 30 07:01:59.897221 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.897202 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:01:59.916933 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.916888 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj"] Oct 30 07:01:59.923252 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.923225 5001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5nqnt\"/\"default-dockercfg-w6ccv\"" Oct 30 07:01:59.935585 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.935549 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hbpn7_ba380ad4-2195-4cfc-b197-d1568adc5140/manager/0.log" Oct 30 07:01:59.941658 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.941616 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-sys\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:01:59.941856 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.941699 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-podres\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:01:59.941856 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.941746 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxcwv\" (UniqueName: \"kubernetes.io/projected/6ed4fdad-7195-478b-8f3c-515faf02676e-kube-api-access-nxcwv\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:01:59.941856 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.941840 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-proc\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:01:59.942623 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:01:59.941927 5001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-lib-modules\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.012390 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.012349 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5nqnt\"/\"openshift-service-ca.crt\"" Oct 30 07:02:00.012594 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.012393 5001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5nqnt\"/\"kube-root-ca.crt\"" Oct 30 07:02:00.035149 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.035110 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d"] Oct 30 07:02:00.039563 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.039534 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2f6e7fc-dnk5d"] Oct 30 07:02:00.042805 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.042765 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-proc\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.042970 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.042834 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-lib-modules\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.042970 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.042916 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-proc\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.042970 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.042922 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-sys\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.043132 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.042988 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-lib-modules\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.043132 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.042993 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-podres\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.043132 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.043028 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-sys\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.043132 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.043043 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ed4fdad-7195-478b-8f3c-515faf02676e-podres\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.043132 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.043048 5001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxcwv\" (UniqueName: \"kubernetes.io/projected/6ed4fdad-7195-478b-8f3c-515faf02676e-kube-api-access-nxcwv\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.066879 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.066845 5001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxcwv\" (UniqueName: \"kubernetes.io/projected/6ed4fdad-7195-478b-8f3c-515faf02676e-kube-api-access-nxcwv\") pod \"perf-node-gather-daemonset-mhpsj\" (UID: \"6ed4fdad-7195-478b-8f3c-515faf02676e\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.117024 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.116988 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07561896-3599-4430-adcf-02bff0a0b23b" path="/var/lib/kubelet/pods/07561896-3599-4430-adcf-02bff0a0b23b/volumes" Oct 30 07:02:00.171165 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.171073 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-66g66_3ed04332-3bea-4244-9a83-4976f78ee337/limitador/0.log" Oct 30 07:02:00.212775 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.212738 5001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:00.357222 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.357187 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-thdmg_8f6114c0-230e-40ef-9cb5-60f49d0f4825/manager/0.log" Oct 30 07:02:00.531882 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:00.531789 5001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj"] Oct 30 07:02:00.534949 ip-10-0-128-31 kubenswrapper[5001]: W1030 07:02:00.534910 5001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6ed4fdad_7195_478b_8f3c_515faf02676e.slice/crio-d8252c51389daecbc477bc73c248213a7a5419d7c5bb451ffe0ec682e2d59dca WatchSource:0}: Error finding container d8252c51389daecbc477bc73c248213a7a5419d7c5bb451ffe0ec682e2d59dca: Status 404 returned error can't find the container with id d8252c51389daecbc477bc73c248213a7a5419d7c5bb451ffe0ec682e2d59dca Oct 30 07:02:01.296945 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:01.296866 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" event={"ID":"6ed4fdad-7195-478b-8f3c-515faf02676e","Type":"ContainerStarted","Data":"d6407aaf3ebd30c599b5be8a6d7c8281e8f81036e1b9d9a470d688934dcbbf6a"} Oct 30 07:02:01.296945 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:01.296933 5001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" event={"ID":"6ed4fdad-7195-478b-8f3c-515faf02676e","Type":"ContainerStarted","Data":"d8252c51389daecbc477bc73c248213a7a5419d7c5bb451ffe0ec682e2d59dca"} Oct 30 07:02:01.297716 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:01.297050 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:01.336615 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:01.336561 5001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" podStartSLOduration=2.336547519 podStartE2EDuration="2.336547519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 07:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:01.335473088 +0000 UTC m=+4135.331430101" watchObservedRunningTime="2026-10-30 07:02:01.336547519 +0000 UTC m=+4135.332504529" Oct 30 07:02:02.717140 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:02.717108 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-gkktj_42155cb9-5b97-4aa0-8a18-70af60804e34/cluster-monitoring-operator/0.log" Oct 30 07:02:04.164823 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:04.164746 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjstw_34b1a113-68c8-4b8c-a46f-52b4c63d11e6/node-exporter/0.log" Oct 30 07:02:04.281252 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:04.281220 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjstw_34b1a113-68c8-4b8c-a46f-52b4c63d11e6/kube-rbac-proxy/0.log" Oct 30 07:02:04.398321 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:04.398282 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjstw_34b1a113-68c8-4b8c-a46f-52b4c63d11e6/init-textfile/0.log" Oct 30 07:02:07.314183 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:07.314151 5001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-mhpsj" Oct 30 07:02:08.232125 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:08.232093 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t_cea443fd-b42f-4897-9731-8354eb3ca6db/istio-proxy/0.log" Oct 30 07:02:08.263581 ip-10-0-128-31 kubenswrapper[5001]: E1030 07:02:08.263540 5001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtvjb" containerName="extractor" Oct 30 07:02:08.883984 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:08.883948 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-5twq8_d6d3694c-d99c-43d1-be3f-6df16717e1de/networking-console-plugin/0.log" Oct 30 07:02:10.034386 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:10.034340 5001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx"] Oct 30 07:02:10.041465 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:10.041432 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t_cea443fd-b42f-4897-9731-8354eb3ca6db/istio-proxy/0.log" Oct 30 07:02:10.045865 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:10.045842 5001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c011337f-b7gkx"] Oct 30 07:02:10.118752 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:10.118713 5001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3364fda6-e33a-4ada-abfa-84cba1506d16" path="/var/lib/kubelet/pods/3364fda6-e33a-4ada-abfa-84cba1506d16/volumes" Oct 30 07:02:10.186831 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:10.186801 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/1.log" Oct 30 07:02:10.193855 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:10.193828 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7ztmq_cb9c2d4d-67e5-4c4d-8294-86ae9050232a/console-operator/2.log" Oct 30 07:02:10.723804 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:10.723770 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-scslz_e1d1dda1-333e-4206-9933-ffe9acdc95c7/discovery/0.log" Oct 30 07:02:10.890459 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:10.890427 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-5nt9s_aed4e7ce-ff6a-4e62-92da-33fc1092c705/istio-proxy/0.log" Oct 30 07:02:11.031467 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:11.031383 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-s6zsz_ff80c97c-2e4d-402f-88a3-4168cc366e05/istio-proxy/0.log" Oct 30 07:02:11.093535 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:11.093503 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7_7a158a21-1de7-4e61-bfda-d255713a8655/istio-proxy/0.log" Oct 30 07:02:11.297153 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:11.297073 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b994dfc85-2v9ww_84346da8-55f2-4344-b7d0-dbfb2996f2e7/router/0.log" Oct 30 07:02:12.498631 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:12.498594 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-scslz_e1d1dda1-333e-4206-9933-ffe9acdc95c7/discovery/0.log" Oct 30 07:02:12.713350 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:12.713293 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-5nt9s_aed4e7ce-ff6a-4e62-92da-33fc1092c705/istio-proxy/0.log" Oct 30 07:02:12.772046 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:12.771960 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-2ql6w_ac9c2fad-cc58-4ec5-b169-271b6937af4b/volume-data-source-validator/0.log" Oct 30 07:02:12.850884 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:12.850855 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-s6zsz_ff80c97c-2e4d-402f-88a3-4168cc366e05/istio-proxy/0.log" Oct 30 07:02:12.917684 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:12.917655 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7_7a158a21-1de7-4e61-bfda-d255713a8655/istio-proxy/0.log" Oct 30 07:02:13.054485 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:13.054391 5001 scope.go:122] "RemoveContainer" containerID="9a9c7a5a37ce2b163516b7ce18cf467b375928480dd5b03a6bb244e5224c00dd" Oct 30 07:02:13.139934 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:13.139906 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b994dfc85-2v9ww_84346da8-55f2-4344-b7d0-dbfb2996f2e7/router/0.log" Oct 30 07:02:14.801279 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:14.801248 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfrtd7t_cea443fd-b42f-4897-9731-8354eb3ca6db/istio-proxy/0.log" Oct 30 07:02:15.368059 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:15.368026 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5gv9v_800e718b-7419-4df9-836a-cea3686ceda9/dns/0.log" Oct 30 07:02:15.482323 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:15.482269 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5gv9v_800e718b-7419-4df9-836a-cea3686ceda9/kube-rbac-proxy/0.log" Oct 30 07:02:16.233391 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:16.233359 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dz874_7212e5ac-eea5-438a-a162-74821cf38b76/dns-node-resolver/0.log" Oct 30 07:02:17.246936 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:17.246899 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4cbf4fd4-7wnrx_383e8617-b682-40d9-9ad6-8f5be7ba56d2/authorino/0.log" Oct 30 07:02:17.290020 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:17.289976 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9s8vq_937866c0-656c-45fb-bec7-b19124c11c4e/manager/0.log" Oct 30 07:02:17.357683 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:17.357653 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-g5mql_f611b837-f97a-4ff0-bc4c-3119f5a8d9ca/manager/0.log" Oct 30 07:02:17.461405 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:17.461369 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-scslz_e1d1dda1-333e-4206-9933-ffe9acdc95c7/discovery/0.log" Oct 30 07:02:17.595735 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:17.595612 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hbpn7_ba380ad4-2195-4cfc-b197-d1568adc5140/manager/1.log" Oct 30 07:02:17.658054 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:17.658026 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-5nt9s_aed4e7ce-ff6a-4e62-92da-33fc1092c705/istio-proxy/0.log" Oct 30 07:02:17.709684 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:17.709643 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l652g_f7ce5e27-0a29-46db-a45d-94dd98cbe059/node-ca/0.log" Oct 30 07:02:17.789569 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:17.789534 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-s6zsz_ff80c97c-2e4d-402f-88a3-4168cc366e05/istio-proxy/0.log" Oct 30 07:02:17.852712 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:17.852624 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-lmck7_7a158a21-1de7-4e61-bfda-d255713a8655/istio-proxy/0.log" Oct 30 07:02:18.006099 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:18.006063 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hbpn7_ba380ad4-2195-4cfc-b197-d1568adc5140/manager/0.log" Oct 30 07:02:18.083981 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:18.083954 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b994dfc85-2v9ww_84346da8-55f2-4344-b7d0-dbfb2996f2e7/router/0.log" Oct 30 07:02:18.087564 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:18.087537 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-66g66_3ed04332-3bea-4244-9a83-4976f78ee337/limitador/0.log" Oct 30 07:02:18.213408 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:18.213374 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-thdmg_8f6114c0-230e-40ef-9cb5-60f49d0f4825/manager/0.log" Oct 30 07:02:18.671549 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:18.671511 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4cbf4fd4-7wnrx_383e8617-b682-40d9-9ad6-8f5be7ba56d2/authorino/0.log" Oct 30 07:02:18.712635 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:18.712604 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9s8vq_937866c0-656c-45fb-bec7-b19124c11c4e/manager/0.log" Oct 30 07:02:18.780269 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:18.780240 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-g5mql_f611b837-f97a-4ff0-bc4c-3119f5a8d9ca/manager/0.log" Oct 30 07:02:19.046198 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:19.046094 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hbpn7_ba380ad4-2195-4cfc-b197-d1568adc5140/manager/1.log" Oct 30 07:02:19.332162 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:19.332071 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hbpn7_ba380ad4-2195-4cfc-b197-d1568adc5140/manager/0.log" Oct 30 07:02:19.366896 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:19.366867 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-66g66_3ed04332-3bea-4244-9a83-4976f78ee337/limitador/0.log" Oct 30 07:02:19.471415 ip-10-0-128-31 kubenswrapper[5001]: I1030 07:02:19.471381 5001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-thdmg_8f6114c0-230e-40ef-9cb5-60f49d0f4825/manager/0.log"